Pillow: Compilation fails when openjpeg 1.5 and 2.x are both installed

[weatherfrog]

Pillow 4.0.0 install fails with Python 3.6 (it works fine with 3.5).
All non-Python dependencies seem to be installed (since it works with Python 3.5).

OS: FreeBSD 10.3

(venv36) $ pip install pillow==4.0.0
[...]
libImaging/Jpeg2KDecode.c:24:5: error: unknown type name 'OPJ_UINT32'
        OPJ_UINT32 tile_index;
        ^
libImaging/Jpeg2KDecode.c:25:5: error: unknown type name 'OPJ_UINT32'
        OPJ_UINT32 data_size;
        ^
libImaging/Jpeg2KDecode.c:26:5: error: unknown type name 'OPJ_INT32'
        OPJ_INT32  x0, y0, x1, y1;
        ^
libImaging/Jpeg2KDecode.c:27:5: error: unknown type name 'OPJ_UINT32'
        OPJ_UINT32 nb_comps;
[...]
libImaging/Jpeg2KDecode.c:521:12: error: use of undeclared identifier 'OPJ_CLRSPC_GRAY'; did you mean 'CLRSPC_GRAY'?
        { "L", OPJ_CLRSPC_GRAY, 1, j2ku_gray_l },
               ^~~~~~~~~~~~~~~
               CLRSPC_GRAY
/usr/local/include/openjpeg.h:134:2: note: 'CLRSPC_GRAY' declared here
            CLRSPC_GRAY = 2,                /**< grayscale */
            ^
[...]
fatal error: too many errors emitted, stopping now [-ferror-limit=]
    20 errors generated.
[...]
cc: error: no such file or directory: 'build/temp.freebsd-10.3-RELEASE-p11-amd64-3.6/decode.o'
cc: error: no such file or directory: 'build/temp.freebsd-10.3-RELEASE-p11-amd64-3.6/encode.o'
cc: error: no such file or directory: 'build/temp.freebsd-10.3-RELEASE-p11-amd64-3.6/map.o'
cc: error: no such file or directory: 'build/temp.freebsd-10.3-RELEASE-p11-amd64-3.6/display.o'
cc: error: no such file or directory: 'build/temp.freebsd-10.3-RELEASE-p11-amd64-3.6/libImaging/Jpeg2KDecode.o'
cc: error: no such file or directory: 'build/temp.freebsd-10.3-RELEASE-p11-amd64-3.6/libImaging/Jpeg2KEncode.o'
error: command 'cc' failed with exit status 1

[wiredfool]

That really looks like an incompatible version of openjpeg.

$ grep CLRSPC_GRAY /usr/include/openjpeg-2.1/openjpeg.h
    OPJ_CLRSPC_GRAY = 2,        /**< grayscale */

Pillow is known to work with 2.0 and 2.1, but not the 1.5 series as is packaged in some operating systems. What version of openjpeg do you have?

[weatherfrog]

I have openjpeg 2.1.2:

$ pkg info openjpeg
openjpeg-2.1.2_1
Name           : openjpeg
Version        : 2.1.2_1
Installed on   : Mon Nov 28 11:43:36 2016 CET
Origin         : graphics/openjpeg
Architecture   : freebsd:10:x86:64
Prefix         : /usr/local
Categories     : graphics
Licenses       : BSD2CLAUSE
Maintainer     : [email protected]
WWW            : http://www.openjpeg.org/
Comment        : Open-source JPEG 2000 codec
Shared Libs required:
    liblcms2.so.2
    libpng16.so.16
    libtiff.so.5
Shared Libs provided:
    libopenjp2.so.7

I don't think the version of openjpeg is the problem since Pillow does compile on the same system but with Python 3.5 instead of 3.6.

[wiredfool]

The compiler error that you're getting is in the Jpeg2k code, which uses openjpeg as it's backend library.

It's referring to a constant that's defined in openjpeg.h, which appears to be different on freebsd than elsewhere.

This is the upstream line that defines OPJ_CLRSPC_GRAY in 2.1.2:
https://github.com/uclouvain/openjpeg/blob/v2.1.2/src/lib/openjp2/openjpeg.h#L283

       OPJ_CLRSPC_GRAY = 2,     /**< grayscale */

This is the same line from openjpeg 1.5: https://github.com/uclouvain/openjpeg/blob/openjpeg-1.5/libopenjpeg/openjpeg.h:

    CLRSPC_GRAY = 2,        /**< grayscale */

So, you've got the openjpeg1.5 port installed, not openjpeg 2.x.

[weatherfrog]

Thanks for insisting on this. I could finally figure out what's going on. You were right, I have installed both openjpeg1.5 and openjpeg2:

$ locate openjpeg.h
/usr/local/include/openjpeg.h
/usr/local/include/openjpeg-2.1/openjpeg.h

Looking at the output of pip install pillow, I think Pillow compiles against the former (v1.5) openjpeg.h because the include path /usr/local/include comes first:

cc [...] -I/usr/local/include -I/usr/local/include/openjpeg-2.1 [...]

It does compile with manual CFLAGS:

$ CFLAGS="-I/usr/local/include/openjpeg-2.1" pip install pillow

So my problem is "solved".

For some reason, the problem occurs only with Python 3.6. I'm not familiar with the Pillow code base. Do you see a way of fixing this? Maybe here?
https://github.com/python-pillow/Pillow/blob/master/setup.py#L455

[aclark4life]

@weatherfrog If you have changes to setup.py that appear to work, you are welcome to submit a PR. We can probably determine fairly quickly if they are detrimental to other platforms.

[weatherfrog]

ok, I'll see what I can do

[radarhere]

Closing as a duplicate of #2563