Kombu: Cannot serialize a string larger than 4GiB using pickle

[vibhavagarwal5]

Hi, looks like there is a new protocol ie 4 for supporting large files. Can someone send a PR for this.

[thedrow]

What are you talking about exactly?

[vibhavagarwal5]

I got this error while passing a large file as an argument to a @celery.task
kombu.exceptions.EncodeError: cannot serialize a string larger than 4GiB
Turns out if you update the serialization.py with the protocol as 4 this might solve this error.

[thedrow]

But it will also create a problem for older clients.
Also why are you sending so much information?

[vibhavagarwal5]

I was using this for loading the pickles async in background and then use these throughout my application.

[matteius]

Protocol version 4 was added in Python 3.4 and 3.4 was EOL on March 18th of this great year of the 2019 and Python 2 is EOL of this next January 1st of 2020. Seems like something Celery 5 should just use by default is protocol 4.

--

[matteius]

I would say to @vibhavagarwal5 that if this is a must have configurable feature for Celery 4, the default should remain as is and provide a way to override the pickle protocol version in all parts of the codebase -- feel free to submit a patch.