Flower: Error saving to database during response to SIGTERM

[ShaheedHaque]

Sometimes, when Flower is shutdown using SIGTERM, it fails to update its database like this:

$ pkill -f celery
...
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/flower/app.py", line 68, in stop
self.events.stop()
File "/usr/local/lib/python3.6/dist-packages/flower/events.py", line 98, in stop
state['events'] = self.state
File "/usr/lib/python3.6/shelve.py", line 124, in __setitem__
p.dump(value)
RuntimeError: OrderedDict mutated during iteration

This happens perhaps 1 in every 4 times for me. The versions I am running are:

flower -> flower:0.9.2 tornado:4.5.3 babel:2.3.4
software -> celery:4.1.0 (latentcall) kombu:4.1.0 py:3.6.3
billiard:3.5.0.3 py-amqp:2.2.2
platform -> system:Linux arch:64bit, ELF imp:CPython
loader -> celery.loaders.default.Loader
settings -> transport:amqp results:disabled

and the Flower command line looks like this:

/usr/bin/python3 /usr/local/bin/celery flower --address=127.0.0.1 --port=5555 -A myappl --logging=debug --persistent --db=/home/.../celery-flower.db --log-file-prefix=/home/.../celery-flower.log --log-file-max-size=5000000

[zwg-zero]

same problem with flower lower==0.9.2 celery==4.1.0 tornado ==5.0.1 python3.6. googled around and some guys said because after python3.4(not included) ordereddic keys() returned a generator, so trying to change will get "OrderedDict mutated during iteration". I'm not sure if it's related to this error.

[zwg-zero]

I debug this problem, found when do pickle of event there are two levels of orderedDIct in evet, and komubu/utils/functional.py (line 124) method "__getstate__" only converted first level of orderedDic to dic, so when pickle save (maybe changing orderedDict), error raise.
I changed the function like below:

       def __getstate__(self):
        d = dict(vars(self))
        d.pop('mutex')
        for k, v in d.items():
            if type(v) == OrderedDict:
                d[k] = dict(v)
        return d

• note:
  >this work around not been tested if it would impact celery. this change will make the type of self.data in "class LRUCache(UserDict)" from OrderedDic() to dict, so may impact 'LRUCache' work correctly.

[ShaheedHaque]

Oh, nice catch!

[zwg-zero]

the reason of the error is orderedDic (self.data keeping tasks) in class 'LRUCache' in flower keeping changing during the pickle progress, so if we want to fix this bug in flower, we have to found a way to stop changing of tasks, but I can't find a way, so
finally I changed the code in kombu/utils/functional.py 'LURCache' class like below.
changed both function of __getstate__ and __setstate__ to avoid impacting other code (keep the order of items in LURCache)
the logic is to convert orderedDic of data into a list of tuple (k,v of orderedDict items) when before pickle, and when depickle, convert list of tuple into orderedDIct.

    def __getstate__(self):
        with self.mutex:
            d = dict(vars(self))
            d.pop('mutex')
            for k, v in d.items():
                if k == 'data' and type(v) == OrderedDict:
                    d[k] = [(in_k, in_v) for in_k, in_v in v.items()]
            return d

    def __setstate__(self, state):
        for k, v in state.items():
            if k == 'data' and type(v) == list:
                state[k] = OrderedDict(v)
        self.__dict__ = state
        self.mutex = threading.RLock()

[yogevyuval]

@mher Any update on this one? flower is not persistent because of that :(

[Pjack]

It's a critical issue for us too.
Once we restart the process, the data lost.
I hope this issue can be fixed in next version.
This!

[sshishov]

Issue will be fixed with celery >= 4.4.0 version

[mher]

Works with celery >= 4.4.0