rq-py3.4 not compatible with rq-py2.7 due to pickle protocol version

Created on 3 Nov 2015  路  4Comments  路  Source: rq/rq

I am playing around with workers with different Python versions. One of the pitfalls is the pickle protocol version used. For python3.4 rq uses the highest available protocol version (4) but python2.7 only understand up to 2. This can be fixed by using a custom Job class and forcing python3.4 to use protocol version 2 but it's rather awkward.
I managed to use a custom Job class (overwriting the dumps method) to overcome this issue but exposing this as a flag might be more convenient.

Most helpful comment

As a side note, while not particularly pretty, this appears to actually work:

import pickle
pickle.HIGHEST_PROTOCOL = 2
from rq import Queue 
# from here on RQ will use pickle protocol 2

At least when using python 3.4 to queue the job, and 2.7 to read it.

All 4 comments

:+1:

There's an issue for implementing a customizable args/kwargs serializer. I think that's the better approach for this particular problem :)

Here's a link to the issue I mentioned: https://github.com/nvie/rq/issues/369 . I forgot to attach it when I closed this issue, sorry.

As a side note, while not particularly pretty, this appears to actually work:

import pickle
pickle.HIGHEST_PROTOCOL = 2
from rq import Queue 
# from here on RQ will use pickle protocol 2

At least when using python 3.4 to queue the job, and 2.7 to read it.

Was this page helpful?
0 / 5 - 0 ratings

Related issues

anseljh picture anseljh  路  3Comments

guhcampos picture guhcampos  路  8Comments

zt9 picture zt9  路  5Comments

mitakuye picture mitakuye  路  7Comments

EliFinkelshteyn picture EliFinkelshteyn  路  7Comments