Pillow: OverflowError in PIL.Image.fromarray

[birgander2]

When trying o generate a huge image (73067 x 28160) from a numpy ndarray, pillow throws me an
"OverflowError: size does not fit in an int". Is pillow limited to maximum sizes of below 2**31bit?

I used:

im = PIL.Image.fromarray(array, mode='F')

[wiredfool]

There are some tests for large images Tests/large_memory_numpy_test.py, but the most they've gone up to is just over 2 gigapixel.

[hugovk]

@birgander2 What's the full trace?

[birgander2]

    arr = PIL.Image.fromarray(array[k, ...].astype(np.float32), mode='F')
  File "/opt/anaconda/lib/python3.4/site-packages/PIL/Image.py", line 2163, in fromarray
    return frombuffer(mode, size, obj, "raw", rawmode, 0, 1)
  File "/opt/anaconda/lib/python3.4/site-packages/PIL/Image.py", line 2113, in frombuffer
    return frombytes(mode, size, data, decoder_name, args)
  File "/opt/anaconda/lib/python3.4/site-packages/PIL/Image.py", line 2047, in frombytes
    im.frombytes(data, decoder_name, args)
  File "/opt/anaconda/lib/python3.4/site-packages/PIL/Image.py", line 728, in frombytes
    s = d.decode(data)
OverflowError: size does not fit in an int
>>> 

And I have to correct myself, the array has only shape (36352, 18144)

[radarhere]

From my tests, here is the minimal code required to produce the problem -

import numpy
from PIL import Image

array = numpy.ndarray(shape=(16384, 16384))
Image.fromarray(array)

So the limit is not 2 ** 31, it's 2 ** 28.

[wiredfool]

Can you try running the large memory test listed above?

[radarhere]

Both of the large memory tests pass for me.

[wiredfool]

@birgander2 What platform are you on, and are you on a 64 bit python?

[birgander2]

I'm on linux using python 3.4.3 with pillow 3.0.0. And yes, it is 64bit python.

[wiredfool]

These tests pass slowly:

    def _write_png(self, xdim, ydim):
        dtype = np.uint8
        a = np.zeros((xdim, ydim), dtype=dtype)
        f = self.tempfile('temp.png')
        im = Image.fromarray(a, 'L')
        im.save(f)

    def test_large(self):
        """ succeeded prepatch"""
        self._write_png(XDIM, YDIM)

    def test_2gpx(self):
        """failed prepatch"""
        self._write_png(XDIM, XDIM)

    def test_1475(self):
        """ issue #1475 """
        self._write_png(36352, 18144)

This test fails:

        a = np.zeros((36352, 18144), dtype=np.float)
        im = Image.fromarray(a, mode='F')

I'm digging in to see where we're actually failing, but it does appear that the float has something to do with it.

[wiredfool]

Yep. There's an int length in there.

What's happening is that for some cases, we need to decode the bytes.

That's effectively here in decode.c:

    UINT8* buffer;
    int bufsize, status;
    ImagingSectionCookie cookie;

    if (!PyArg_ParseTuple(args, PY_ARG_BYTES_LENGTH, &buffer, &bufsize))
        return NULL;

It's an integer bufsize. It looks like it can be fixed by changing all the ints to py_ssize_t, and defining PY_SSIZE_T_CLEAN before including Python.h

[ibesora]

Hi, is there some progress in this issue?
I've been working with very big images and I get the same error on version 5.

[wiredfool]

No progress.

[ibesora]

Thanks for the info @wiredfool . I suppose it's not something with high priority. Will resort to a workaround instead.

[rcasero]

I think I'm having this problem too, when trying to read this image, size (51200, 38144) ~ 1.82 Gpixels,

http://openslide.cs.cmu.edu/download/openslide-testdata/Hamamatsu/CMU-1.ndpi

with OpenSlide. The code

import openslide
slide = openslide.OpenSlide("CMU-1.ndpi")
foo = slide.read_region(location=(0, 0), level=0, size=slide.dimensions)

gives the error

Traceback (most recent call last):
  File "<input>", line 4, in <module>
  File "/home/rcasero/.conda/envs/elastixity/lib/python3.6/site-packages/openslide/__init__.py", line 223, in read_region
    level, size[0], size[1])
  File "/home/rcasero/.conda/envs/elastixity/lib/python3.6/site-packages/openslide/lowlevel.py", line 260, in read_region
    return _load_image(buf, (w, h))
  File "/home/rcasero/.conda/envs/elastixity/lib/python3.6/site-packages/openslide/lowlevel.py", line 65, in _load_image
    return PIL.Image.frombuffer('RGBA', size, buf, 'raw', 'RGBA', 0, 1)
  File "/home/rcasero/.conda/envs/elastixity/lib/python3.6/site-packages/PIL/Image.py", line 2398, in frombuffer
    core.map_buffer(data, size, decoder_name, None, 0, args)
MemoryError: Integer overflow in ysize

Package versions:

python                    3.6.4                hc3d631a_1
openslide-python          1.1.1                     <pip>
Pillow                    5.1.0                     <pip>

[bgilbert]

@rcasero That may be a different issue; it's at least a new/different code path. Please file a bug here.

[rcasero]

@bgilbert , done, posted to https://github.com/openslide/openslide-python/issues/31

[radarhere]

@wiredfool's suggestion was coincidentally implemented in #3749. With that PR, the error here changes from 'OverflowError: size does not fit in an int' to 'ValueError: not enough image data'.

I have created PR #3791 to take the last step to resolve this.