Rq: More versatile pickling with dill

[willvousden]

dill extend's Python's pickling capabilities to handle things like lambdas and instance methods.

It would be nice if RQ were able to use this, if it's available on the system. As far as I know, it's a drop-in replacement for cPickle.

[selwin]

I try to avoid adding dependencies to RQ, preferring to keep it lightweight. However, we can support if by adding a serializer argument. So for example:

queue = Queue('default', connection=redis, serializer=DillSerializer)

This Serializer object would have serialize and deserialize methods that will be called when writing to/ reading from Redis.

What do you think?

[willvousden]

You're right – probably best to provide it as an argument.

However, it would be nice just to use the conventional dumps/loads module-level functions. This way you could just pass the module itself in with no extra code:

import dill
queue = Queue('default', connection=redis, serializer=dill)

[selwin]

That sounds like a good idea. Please feel free to open a PR for this :).

[rizplate]

is marshmallow supported for the rq redis-configs or job/worker params?

[cipri-tom]

has there been any progress on this? If not, where should one start? Just look for calls to pickle and use the configured serializer instead?

[selwin]

There's been no activity around this issue. The majority of changes should happen around job.restore() and job.to_dict().

Aside from that, Queue and Worker objects must also know what serializer they should use.

[woodruffw]

Opened #1141 as one potential approach to this.