Pillow: Loading pickled images fails for png files with iTXt chunks

[sangaline]

Trying to load a pickled image fails if it's a png file with iTXt chunks. The iTXt.__new__(cls, text, lang, tkey) method appears to be called with only two arguments when Image.__setstate__ is called. All of the test_pickle.py tests pass but I don't think any of those images include iTXt chunks.

Code to reproduce the issue:

from PIL import Image
import pickle

im = Image.open('example.png')
p = pickle.dumps(im)
new_im = pickle.loads(p)

The output of running that code:

Traceback (most recent call last):
  File "example.py", line 6, in <module>
    new_im = pickle.loads(p)
TypeError: __new__() missing 2 required positional arguments: 'lang' and 'tkey'

The image that was being used:

[hugovk]

The example works as expected for me, on OS X with Python 2.7.9, and the contents of each image look the same using .show().

What Python version and OS are you using?

[sangaline]

I get the error on Linux with Python 3.4.3.

[hugovk]

What do you get if you edit test_pickle.py like this and run python Tests/test_pickle.py?

It passed on Travis CI (Linux) on all available Pythons, including 3.4.2:
https://travis-ci.org/hugovk/Pillow/builds/82614673

If it fails for you, please could you also try with the image that was attached here, just in case something changed when uploading to GitHub.

[sangaline]

The additional test passes for me. It seems that the error only occurs when pickle protocol version 2 is being used and the edit you posted only checks the new file with protocol 0. If you change the default file in helper_pickle_string to point to the file with the iTXt chunks then do you get failed tests?

[hugovk]

Thanks, I can reproduce it now with https://github.com/python-pillow/Pillow/commit/9d5864c673302a23f38402b2e7d7f8254e513899, both locally and on Travis CI.

If we print out the class and arguments in pickle.py just before where the TypeError occurs:

    def load_newobj(self):
        args = self.stack.pop()
        cls = self.stack[-1]
        print("cls", cls)
        print("args", args)
        obj = cls.__new__(cls, *args)
        self.stack[-1] = obj
    dispatch[NEWOBJ] = load_newobj

We get:

('cls', <class 'PIL.PngImagePlugin.PngImageFile'>)
('args', ())
('cls', <class 'PIL.PngImagePlugin.iTXt'>)
('args', ('Created with GIMP',))
...
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1085, in load_newobj
    obj = cls.__new__(cls, *args)
TypeError: __new__() takes exactly 4 arguments (2 given)

So it's already created a PNG image object, but is failing when trying to create an iTXt object, presumably because class iTXt(str) has def __new__(cls, text, lang, tkey):...

By the way, is it okay to include the test image in the repo under the PIL, MIT-like licence?

[sangaline]

I'd be honored, please do :-).

Thanks for all of the work you guys do on Pillow by the way, it's an incredibly useful library.

[aclark4life]

:+1:

[Andarius]

Same error using Python 3.5.1 (Linux), PIL version: 1.1.7. Temporary fixed by setting the protocol to 0.

[aclark4life]

Still an open bug issue?

[ubershmekel]

This happens on Python 3.6 on Windows 10. PIL.VERSION '1.1.7'

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\...\Anaconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\...\Anaconda3\lib\multiprocessing\spawn.py", line 115, in _main
    self = reduction.pickle.load(from_parent)
TypeError: __new__() missing 2 required positional arguments: 'lang' and 'tkey'

[Andarius]

Still a problem with PIL '4.1.1'.

[aclark4life]

@Andarius How about 4.2.0? I suspect so… but want to confirm.

[Andarius]

@aclark4life Tried with 4.2.1, still the same.

[aclark4life]

@Andarius Thanks

[jayking0912]

if i use protocol version without 2
TypeError: __new__() missing 2 required positional arguments: 'lang' and 'tkey'

someone who can fix this bug?

[jekbradbury]

This is still an issue on Pillow 5.1. I believe it can be fixed by adding a __getnewargs__ method to the iTXt class, but if you have an existing pickle file you want to load (as I did) that's not going to help. Instead you can use this horrible monkeypatch:

import pickle
pickle.Unpickler = pickle._Unpickler
def load_newobj(self):
    args = self.stack.pop()
    cls = self.stack.pop()
    try:
        obj = cls.__new__(cls, *args)
    except Exception:
        obj = cls.__new__(cls, *args, 'EN', *args)
    self.append(obj)
pickle._Unpickler.load_newobj = load_newobj
pickle._Unpickler.dispatch[pickle.NEWOBJ[0]] = load_newobj

[shanness]

Is there any workaround for this? I'm multiprocessing (basically building a zip and watermarking images in it), and it's failing if the PNG has iTXt chunks. And due to the async nature, I don't think I can use the horribly monkey patch above, as the unpickler happens in the multiprocessing library..

This has broken a pretty important launch we've just done for some customers. Any help much appreciated..

File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run self._target(*self._args, **self._kwargs) File "/usr/lib/python3.6/multiprocessing/pool.py", line 108, in worker task = get() File "/usr/lib/python3.6/multiprocessing/queues.py", line 337, in get return _ForkingPickler.loads(res) TypeError: __new__() missing 2 required positional arguments: 'lang' and 'tkey'

[jekbradbury]

Maybe try patching the multiprocessing library code with the same monkeypatch, if you’re able to modify it?

[shellvon]

@jekbradbury
I also added this line of code(by adding a __getnewargs__ method ). Although this is not well written, but it is the easiest way.

PngImagePlugin.iTXt.__getnewargs__ = lambda x:(x.__str__(), '', '')

[radarhere]

I have created PR #3537 as a possible solution to this.

[gfulton96]

@radarhere I am still receiving the "TypeError: __new__() missing 2 required positional arguments: 'lang' and 'tkey'" Error with PIL version 5.4.1. Any fixes? :)

[radarhere]

The fix is not a part of 5.4.1. It will be a part of 6.0.0, to be released on April 1.

[radarhere]

You can apply the changes from the PR to your own installation manually, going into your Pillow installation and changing the file - https://github.com/python-pillow/Pillow/pull/3537/files#diff-07cbf160cd247acb9b67060ff3d81ed9

commit f8df6d1687c2eba376b03b60a59b63c390c7d810
Author: Andrew Murray <[email protected]>
Date:   Wed Jan 2 18:56:54 2019 +1100

    Fixed pickling of iTXt class with protocol > 1

diff --git a/src/PIL/PngImagePlugin.py b/src/PIL/PngImagePlugin.py
index 04161a56c..57d2f8341 100644
--- a/src/PIL/PngImagePlugin.py
+++ b/src/PIL/PngImagePlugin.py
@@ -192,7 +192,7 @@ class iTXt(str):

     """
     @staticmethod
-    def __new__(cls, text, lang, tkey):
+    def __new__(cls, text, lang=None, tkey=None):
         """
         :param cls: the class to use when creating the instance
         :param text: value for this key

[gfulton96]

Awesome thanks @radarhere. It works :)

[radarhere]

@gfulton96 Pillow 6.0.0 has now been released.