Pillow for Python 3.7 on FreeBSD installs but doesn't work

[igoose1]

What did you do?

I installed Pillow by pip3.7 and, after this way didn't fix an error, I compiled from source. After I'm trying to launch tests (python3.7 selftest.py).

What actually happened?

Library is installing well but when I'm trying to import Image it fails with

>>> from PIL import Image
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/user/.local/lib/python3.7/site-packages/PIL/Image.py", line 90, in <module>
    from . import _imaging as core
ImportError: Shared object "libz.so.1" not found, required by "_imaging.so"

That's also an error in tests (not suprisingly):

[user@pc][~/.../Pillow-6.2.1]% python3.7 selftest.py
Traceback (most recent call last):
  File "selftest.py", line 8, in <module>
    from PIL import Image, features
  File "/home/user/.local/lib/python3.7/site-packages/PIL/Image.py", line
90, in <module>
    from . import _imaging as core
ImportError: Shared object "libz.so.1" not found, required by "_imaging.so"

What are your OS, Python and Pillow versions?

• OS: FreeBSD 12.0
• Python: Python 3.7.5 (or ipython 7.7.0)
• Pillow: 6.2.1 with pip and from source on PyPI.

---

To notice, previous versions of Python (e.g. Python 3.6) don't work by simple pip install Pillow correctly too. For that case there are ports for py36 and py27 but no ports for other versions: https://www.freshports.org/graphics/py-pillow.

I cannot understand how maintainers fix these errors as I'm not good at that but I think this commit can help: https://svnweb.freebsd.org/ports?view=revision&revision=508031.

[igoose1]

@radarhere, could you please explain why did you set "Linux" label? I'm talking about FreeBSD which is not a Linux distributive, it's an another OS with different rules, licensing views and another kernel.

[igoose1]

Btw, there is a special "FreeBSD" label for that.

[hugovk]

Thanks for the report.

We don't explicitly support FreeBSD and point people towards FreshPorts:

https://pillow.readthedocs.io/en/stable/installation.html#freebsd-installation

It's best to contact the official ports team directly. Looks like @koobs is the maintainer of the py-pillow port (thank you for your work!).

[koobs]

Hi @igoose1 (and thanks for the heads up @hugovk)

The TLDR is:

This report is not a FreeBSD, or Ports/Package specific issue (pip installation into a homedir was used), and a likely bug in upstream build, modulo any unexpected system/environment issues. Extremely common class of issue with C extension based Python builds on many OS's.

This Shared object "libz.so.1" not found error is likely either: pillow not finding the base zlib headers/libraries, OR its finding the base libraries, but its also finding a port/package installed zlib headers/libraries, and conflicting. Its difficult to say without more system and pillow build/compile information.

Nevertheless, I'd recommend using FreeBSD Ports/Packages where available because we've done the "special casing" and "porting" that's often needed to build software, especially those that use third party libraries.

Ports you can build for any Python version, precompiled Packages are available for the two 'major' 'default' Python versions (See below for details)

Long version:

So there's a few issues here:

1) Using the FreeBSD ports/packages vs pip installing
2) Building Python packages (that build C extensions using third party libraries)
3) 2.7 and 3.6 versions of FreeBSD Packages

Note: In this issue report, a standard pip install was used, and is not related to FreeBSD or ports/packages 'perse', so bug reports about the port are not appropriate in this case.

Lets go through each of them:

1) Note: I see from your report that you used pip to install Pillow into your home directory (which is fine), so the following is only for reference:

While pip is provided in FreeBSD ports/packages, it is not supported as a tool for installing packages into the FreeBSD system-wide Python environment. You are free however to use pip's --user flag to install packages into other places like your home directory or into virtual environments as you please. There is a post-installation message to this effect when installing the pip port or package on FreeBSD:

pip MUST ONLY be used:

 * With the --user flag, OR
 * To install or manage Python packages in virtual environments

Failure to follow this warning can and will result in an inconsistent
system-wide Python environment (LOCALBASE/lib/pythonX.Y/site-packages) and
cause errors.

Avoid using pip as root unless you know what you're doing.

This is the best balance we can strike between providing default and expected Python tools to our users, and keeping their systems safe.

It's also worth noting that when one is installing things outside of the standard system locations, such as your home directory, that it can be even harder to get software to build without passing your custom include/library locations to the software build, which is the root cause of basically 80% of Python C extension failures that I've ever seen.

2) Most software packages, including Python packages providing C extensions, if they do so at all, tend to only look in "standard" include/library directories, which most of the time means Linux's /usr/lib & /usr/include and MacOS locations, unless they use pkg-config / autotools. FreeBSD installs third party libraries and headers in /usr/local/include and /usr/local/lib by default (as it separates base and third party software), which most often means users have to pass custom CFLAGS/LDFLAGS to software builds so that the build systems find the relevant files.

In this case, Pillow actually does more than most, and has, (granted, very complex) routines (see setup.py) which do the following:

        # standard locations
        if not self.disable_platform_guessing:
            _add_directory(library_dirs, "/usr/local/lib")
            _add_directory(include_dirs, "/usr/local/include")

            _add_directory(library_dirs, "/usr/lib")
            _add_directory(include_dirs, "/usr/include")

As well as some freebsd specific bits (contrary to any general 'freebsd is not supported' comment):

def _find_library_dirs_ldconfig():
...
    elif sys.platform.startswith("freebsd"):
        args = ["/sbin/ldconfig", "-r"]
...
            or sys.platform.startswith("freebsd")
        ):
            for dirname in _find_library_dirs_ldconfig():
                _add_directory(library_dirs, dirname)

To help it find the files it needs.

3) Only seeing py27-pillow and py36-pillow FreeBSD Packages

• A FreeBSD Port (think "source package") can be built for any version of Python that FreeBSD supports (currently 2.7,3.5-3.8) that the software also supports (Pillow is not restricted). Its a simple:

cd /usr/ports/graphics/py-pillow && FLAVOR=py38 make install clean

To build Pillow for and with Python 3.8

You can even install multiple concurrent versions of the same software for multiple Python versions.

For FreeBSD Packages (Note: Packages are literally just pre-compiled Ports), we only build binary packages for the two default (major) versions of Python, which at this time are 2.7 and 3.6. This is why you only see two.

[igoose1]

Thank you for the answer, @koobs! You've cleared a lot for me.

I realised that I have to use ports of libraries (or pip install in home directory or virtual environment if there are no ports). Unfortunately in that case I couldn't have found pillow of py38. make(1) stops with an error that there're just 2 possible flavors (py36 and py27) but I think it's the problem of mine so issue can be closed.

@hugovk, everything I needed was discussed and I got an answer for problems of installing python packages in FreeBSD. Thank you! Issue can be closed.

[hugovk]

Thanks both!

[koobs]

Thank you for the answer, @koobs! You've cleared a lot for me.

My pleasure, you're welcome.

I realised that I have to use ports of libraries (or pip install in home directory or virtual environment if there are no ports). Unfortunately in that case I couldn't have found pillow of py38. make(1) stops with an error that there're just 2 possible flavors (py36 and py27) but I think it's the problem of mine so issue can be closed.

You can use BUILD_ALL_PYTHON_FLAVORS=yes in /etc/make.conf to enable all possible (python version) flavors