Pillow: OJPEGPostDecode: Assertion `sp->libjpeg_session_active!=0' failed

[alexkiro]

While trying to save a, what seems to be broken, TIFF image as PPM the python interpreter crashes with the following error:

>>> from PIL import Image
>>> image = Image.open(open("./bad.tiff"))
>>> image.save(open("/tmp/bad.ppm", "wb"), "PPM")
python: tif_ojpeg.c:848: OJPEGPostDecode: Assertion `sp->libjpeg_session_active!=0'    failed.
Aborted

This is happening with Python2.7.6 and Pillow2.4.0 running on Debian 6.

Please let me know if I can provide you with any more information.

[hugovk]

What happens without the extra open calls?

>>> from PIL import Image
>>> image = Image.open("./bad.tiff")
>>> image.save("/tmp/bad.ppm", "PPM")

[alexkiro]

Same thing:

>>> from PIL import Image
>>> image = Image.open("./bad.tiff")
>>> image.save("/tmp/bad.ppm", "PPM")
python: tif_ojpeg.c:848: OJPEGPostDecode: Assertion `sp->libjpeg_session_active!=0' failed.
Aborted

[hugovk]

OK. Can you check if it works with this file?
https://github.com/python-imaging/Pillow/blob/master/Tests/images/pil136.tiff

It does for me. If not, are you able to share this (or another) problematic image file?

[alexkiro]

It does work with the test file:

>>> from PIL import Image
>>> image = Image.open("pil136.tiff")
>>> image.save("/tmp/bad.ppm", "PPM")
>>> 

I'm afraid I can't share the image here. But I can share it to you in private somehow.

[wiredfool]

1) I'm pretty sure that that's in the self.load() call that's the first part of the save process.
2) If there are jpeg issues in a tiff file, then it's one of the odder compressions -- probably tiff_jpeg, but I seem to remember that there's another deprecated one floating around somewhere.

I'm not sure if you're using libtiff (or even have it compilled in) but it would be worth a try to force libtiff:

from PIL import TiffImagePlugin
TiffImagePlugin.READ_LIBTIFF = True

[alexkiro]

Ok, so more details here then :)

Libraries installed are:

zlib1g-dev libjpeg62-dev libtiff4-dev libfreetype6-dev liblcms1-dev libopenjpeg-dev

While installing Pillow:

--------------------------------------------------------------------
PIL SETUP SUMMARY
--------------------------------------------------------------------
version      Pillow 2.4.0
platform     linux2 2.7.6 (default, Dec 28 2013, 20:51:51)
             [GCC 4.4.5]
--------------------------------------------------------------------
*** TKINTER support not available
--- JPEG support available
*** OPENJPEG (JPEG2000) support not available
--- ZLIB (PNG/ZIP) support available
--- LIBTIFF support available
--- FREETYPE2 support available
*** LITTLECMS2 support not available
*** WEBP support not available
*** WEBPMUX support not available
--------------------------------------------------------------------

Forcing libtiff still results in the same crash:

>>> from PIL import TiffImagePlugin
>>> TiffImagePlugin.READ_LIBTIFF = True
>>> from PIL import Image
>>> image = Image.open("./bad.tiff")
>>> image.save("/tmp/bad.ppm", "PPM")
python: tif_ojpeg.c:848: OJPEGPostDecode: Assertion `sp->libjpeg_session_active!=0' failed.
Aborted

Here is also the a trace using pdb: http://pastebin.com/QaefBUYy

This identified the problem at this line

> /root/se-env/lib/python2.7/site-packages/PIL/TiffImagePlugin.py(722)_load_libtiff()
-> n,err = decoder.decode(b"fpfp") # 4 bytes, otherwise the trace might error out

[wiredfool]

Can you try it with Image.DEBUG=True? I should have asked for that before.

[alexkiro]

I've run this with Image.DEBUG=True. You can find the output here: http://pastebin.com/kn7fNUUX

[wiredfool]

Ok, That's an interesting image. I don't think that I've seen that combination of compression and band format before. It looks like it's from a document management system. Is there any way to scan something non-sensitive and send me the file?

Alternatively, you can email it to me @ eric-github [at] soroos . net

[alexkiro]

By the way I did send you the image via email. Hope that helps.

[wiredfool]

Yes, thanks, I did get it. I've been traveling the last couple days, will look at it soon.

Sent from my iPod

On May 28, 2014, at 7:18 AM, Chirila Alexandru [email protected] wrote:

By the way I did send you the image via email. Hope that helps.

—
Reply to this email directly or view it on GitHub.

[aclark4life]

Targeting for 2.5

[wiredfool]

Sorry, this dropped off my radar.

Imagemagick says (among other things):

identify: bad.tiff: unknown field with tag 37680 (0x9330) encountered. `TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/706.
identify: Depreciated and troublesome old-style JPEG compression mode, please convert to new-style JPEG compression and notify vendor of writing software. `OJPEGSetupDecode' @ warning/tiff.c/TIFFWarnings/706.

Some info on the unknown tag: http://blogs.msdn.com/b/openspecification/archive/2009/12/08/details-of-three-tiff-tag-extensions-that-microsoft-office-document-imaging-modi-software-may-write-into-the-tiff-files-it-generates.aspx -- it's a binary dump of an OLE property set. Which is super helpful, at least we know we can ignore it.

Despite the warnings, imagemagick is able to read and convert the files to png or jpegs, so there should be something that we can do. I'd recommend that as a workaround for now.

[rhettlunn]

To help debugging, here's a publicly available image that seems to have the same issue:
http://www.goresbridgesaddlery.com/wp-content/uploads/2012/10/144612-Korsteel-Loose-Ring-Snaffle-with-Players.tif

[aclark4life]

Would love to see a fix for this in 2.5.0

[aclark4life]

Targeting Future

[radarhere]

From the conversation, I can't see any conclusive statements that this depends on the system involved, but when I try and use the image on OS X from @rhettlunn, under 2.9, I get OSError: cannot write mode YCbCr as PPM and under 2.3.x / 2.5.x I get IOError: -2, neither of which is the reported error.

[rhettlunn]

Ah, interesting - I'm under Ubuntu 12.04.4 - and get the same error when calling image.save() in 2.3.0 as I do in 2.9.0:

python: tif_ojpeg.c:848: OJPEGPostDecode: Assertion `sp->libjpeg_session_active!=0' failed.
Aborted (core dumped)

This is in Python 2.7.3

[jeremyll]

I get the same error message on an image using:

• Ubuntu 12.04
• Python 2.7.3
• Pillow 2.9.0
• openjpeg-2.1.0
• libtiff-4.0.6

[vashek]

I have a very similar error:

Assertion failed: sp->libjpeg_session_active!=0, file tif_ojpeg.c, line 891

Happens on load() of a particular TIFF file. In my case it's on Windows, perhaps that's why the error message is slightly different, including the line number?
I tried Python 2.7.10 32-bit with Pillow 2.9.0, and Python 3.5.2 64-bit with Pillow 3.3.1 -- same result.
PIL.Image.core.jpeglib_version on the Pillow 3.3.1 version is "9.0" (not that I expect it matters, just trying to gather as much info as I can think of). I don't know the libtiff version with which the module was compiled, but Pillow was installed with a simple "pip install pillow".

I can provide the TIFF file publicly, no problem. It seems to be a scan from Xerox WorkCentre 6655. Attached here (in a ZIP): badtiff-pillow-671.zip

Here is the debug log from opening and loading the image, including the source code of how the log was created: http://pastebin.com/UjBkd6rD for the Py3 and http://pastebin.com/wMy45wZe for Py2.
I think it's the same case as above.

How can I help? Is there anything I can do short of actually delving info libtiff (which I'm afraid I don't realistically have time for)?

[vashek]

This http://bugzilla.maptools.org/show_bug.cgi?id=2209 seems like possibly the same thing, but doesn't offer an actual solution - just an error message ("LibJpeg: Improper call to JPEG library in state 0.") instead of a crash, which is certainly an improvement but it still won't load the image from what I understand.

This also mentions the assertion but doesn't have a solution: https://bugzilla.redhat.com/show_bug.cgi?id=583081

And this: http://www.asmail.be/msg0054540514.html / repost in http://www.asmail.be/msg0055066080.html seems to be a patch for libtiff 3.9.5 that someone offered back in 2011, which allegedly should fix (or "workaround", if you prefer) the reading of some broken OJPEG-compressed TIFF files (I don't know if that includes this specific type of brokenness though). As far as I can tell, that patch was never applied unfortunately.

I can't seem to find this problem reported in the libtiff Bugzilla here: http://bugzilla.maptools.org/buglist.cgi?product=libtiff

In case anyone's searching for the current libtiff home, it's here: http://simplesystems.org/libtiff/ and NOT on libtiff.org.

[wiredfool]

The exact line and message are going to vary with the version of libtiff.

Given your research and the location of the assert, it appears to be one of:

1) A slightly malformed tiff
2) a bug in libtiff
3) Some setting that we could workaround.

Of those, I can't say which of 1 or 2 is more likely, but I think they're more likely than 3.

[hugohfsouza]

I have the same problem. I installed several libs, but I did not succeed.

[bzamecnik]

We ran into the same problem with both libtiff-4.0.6 and 4.0.9.

The assertion where it fails is here: https://gitlab.com/libtiff/libtiff/blob/master/libtiff/tif_ojpeg.c#L897

[bzamecnik]

It seems the problem is when loading a TIFF some some old-style JPEG compression. In particular, it was a 4-page TIFF produced by "NEWGEN IMGFL32 2012.10.18 2.85.102.1", with first page BW with CCITT Fax 4 compression and other pages RGB with old-style JPEG compression. Unfortunately I'm not able to publish the actual image file on which it crashes.

Code to crash when loading a broken image:

import PIL.Image
image = PIL.Image.open('bad.tiff')
image.seek(1) # it was second page...
image.tobytes()

This function can test whether the TIFF file contains frames (pages) with old-style JPEG compression that libtiff may crash on:

def is_bad_jpeg_frame(image, frame):
    """
    TIFFs that use 'old-style' JPEG compression can be identified
    from the value '6' (0x0006) of the 'Compression' tag (259/0x0103).

    https://github.com/openpreserve/format-corpus/tree/master/tiff-examples/old-style-jpeg-compression

    They may cause crash of in libtiff.
    """
    image.seek(frame)
    return (image.format == 'TIFF'
        and hasattr(image, 'tag_v2')
        and image.tag_v2.get(259) == 6)

def is_bad_tiff_image(image):
    current_frame = image.tell()
    is_bad = any(is_bad_jpeg_frame(image, frame) for frame in range(image.n_frames))
    image.seek(current_frame)
    return is_bad

[valexby]

Hello, I have same issue as @bzamecnik have. His code for detecting failing TIFFs works well for me:) PIL fails with tif.zip(github don't allow attach tif files, so I've zipped it). Also I'm able to provide a huge number of such failing tiffs, if you need them.

Also could somebody help me to choose set of dependencies, that will open attached tiff without issues?

Currently I'm using:

• Arch 4.18.16
• Python 3.6.5
• Pillow 5.2.0
• openjpeg2 2.3.0-3
• libtiff 4.0.9-2

Thank you!

[kkopachev]

That's happening because TIFFReadScanline for some reason is not able to read OJPEG for some reason.
I had success with using TIFFReadEncodedStrip instead and it worked better, but it returns subsampled data and there is no easy way to unpack it.
Another option would be to use TIFFReadRGBAStrip/TIFFReadRGBATile for all YCbCr data and it will handle subsampling internally. Only downside I see here is that it returns image data in bottom-to-top order and we have to duplicate existing if (tiled) ... else ... block for that case.
/cc @hugovk

[kkopachev]

https://github.com/python-pillow/Pillow/pull/3489 should be able to fix it. I tested it with 2 images from this issue plus image from format-corpus