Python: The python path is not set correctly (does not include `/usr/lib/python2.7/dist-packages/`)

[mapio]

It is well known that the Debian packaged python libraries get installed in a _non-standard_ directory, for instance in /usr/lib/python2.7/dist-packages/ for python2.7 (see Deviations from upstream in the Python Debian wiki).

On the other hand, these images do not install the system provided python, but install it from source so that the sys.path does not take into account the above non-standard directories; for instance, for python2.7, the path is set to

['', '/usr/local/lib/python27.zip', '/usr/local/lib/python2.7', '/usr/local/lib/python2.7/plat-linux2',
 '/usr/local/lib/python2.7/lib-tk', '/usr/local/lib/python2.7/lib-old', '/usr/local/lib/python2.7/lib-dynload',
 '/usr/local/lib/python2.7/site-packages']

If this issue is not solved, it is impossible to use system packaged python libraries; this is particularly critical for libraries that have heavy binary dependencies on other system libraries.

[tianon]

I'm not sure I'm following for what's gained by supporting this path besides encouraging people to mix Debian Python packages with pip install packages? (which is generally considered harmful)

I think if you're relying on Debian-packaged python dependencies, you're probably going to be better off going straight for either FROM debian or FROM buildpack-deps instead of trying to get this image to do what you might expect when mixing dependencies from APT and elsewhere.

[mapio]

I think that, for module with extensions, it is much easier to install a .deb version than using pip – and I don't think it is _generally_ considered harmful to mix them.

Just to make an example, look at the dependencies required to install the mapnik python bindings module… it's such a blessing to be able to use a pre-packaged version!

Moreover, it's also a matter of "least surprise".

The official debian docker image contains the system python, thus behaves differently (let me say "correctly" wrt to this issues), while the python docker, even if derives from the debian image, behaves unexpectedly. It took me quite a while to understand why installing a .deb-ianized python module in debian made it found by the python interpreter, while the same did not happened in the python image.

[rtxanson]

Am currently fixing this in my Dockerfile by just adding dist-packages to PYTHONPATH via an ENV command, but would be nice if this were upstream, potentially others are expecting this behavior and not finding it.

[abe-winter]

numpy and scipy are effectively broken on docker until this is fixed -- pip install scipy doesn't work out of the box. the env tweak from @rtxanson solves this, would be nice to have it by default.

[adriancooney]

I'm having the same problem and @rtxanson fix works for _using_ python however if another dependency needs numpy or scipy (Scikit-Learn for example), it wouldn't work when trying to install with Pip. It was giving this weird error related to distutils when I tried setting the PYTHONPATH to @rtxanson's suggestion. To fix it, I symlinked the numpy and scipy installations to my local site-packages directory then emptied my PYTHONPATH and the fix the broken installations:

RUN bash -c 'ln -s /usr/lib/python2.7/dist-packages/{numpy,scipy}* /usr/local/lib/python2.7/site-packages/'

The weird error I was getting when setting PYTHONPATH to the /usr/lib/python2.7/dist-packages directory for Googlers:

Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-24Xvhq/SQLAlchemy/setup.py", line 170, in <module>
        run_setup(True)
      File "/tmp/pip-build-24Xvhq/SQLAlchemy/setup.py", line 150, in run_setup
        **kwargs
      File "/usr/local/lib/python2.7/distutils/core.py", line 151, in setup
        dist.run_commands()
      File "/usr/local/lib/python2.7/distutils/dist.py", line 953, in run_commands
        self.run_command(cmd)
      File "/usr/local/lib/python2.7/distutils/dist.py", line 972, in run_command
        cmd_obj.run()
      File "/usr/lib/python2.7/dist-packages/setuptools/command/install.py", line 61, in run
        return orig.install.run(self)
      File "/usr/local/lib/python2.7/distutils/command/install.py", line 575, in run
        self.run_command(cmd_name)
      File "/usr/local/lib/python2.7/distutils/cmd.py", line 326, in run_command
        self.distribution.run_command(command)
      File "/usr/local/lib/python2.7/distutils/dist.py", line 971, in run_command
        cmd_obj.ensure_finalized()
      File "/usr/local/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized
        self.finalize_options()
      File "/usr/lib/python2.7/dist-packages/setuptools/command/install_lib.py", line 16, in finalize_options
        self.set_undefined_options('install',('install_layout','install_layout'))
      File "/usr/local/lib/python2.7/distutils/cmd.py", line 302, in set_undefined_options
        getattr(src_cmd_obj, src_option))
      File "/usr/local/lib/python2.7/distutils/cmd.py", line 105, in __getattr__
        raise AttributeError, attr

[tianon]

https://github.com/docker-library/python/issues/88#issuecomment-238387752

I think this is fixed now! :smile:

$ docker run -it --rm python:3 bash
root@a31adf41234d:/# pip install scipy
Collecting scipy
  Downloading scipy-0.18.0-cp35-cp35m-manylinux1_x86_64.whl (42.0MB)
    100% |████████████████████████████████| 42.0MB 58kB/s 
Installing collected packages: scipy
Successfully installed scipy-0.18.0
root@a31adf41234d:/# 

[montefra]

As already said some package are hard/impossible to install with pip. One of them is PyQt{4,5}.
The only alternative to apt that I know is distributions like anaconda.

Now I'm experimenting to get dockers based runners for gitlab-ci and I have a package that needs PyQt.

I can run containers with python:2.7 and 3:5 and install python-qt4 [or python3-pyqt5] with apt-get.

They are installed under /usr/lib/python2.7/dist-packages [/usr/lib/python3/dist-packages/].

Python interpreter in docker don't have those paths, so import PyQt4 fails. I can import them easily using the system python executables.

Update:

• on python2.7: PYTHONPATH=/usr/lib/python2.7/dist-packages:$PYTHONPATH python works as expected.
• on python 3.5: PYTHONPATH=/usr/lib/python3/dist-packages:$PYTHONPATH python3 allows to import PyQt4, but fails with import PyQt4.QtCore

Update#2:

doing an ln -s /usr/lib/python{2.7,3}/dist-packages/{PyQt,sip*} /usr/local/lib/python{2.7,3.5}/site-packages is also an option. However python3.5 import doesn't work as before

Update#3 [hopefully the last]

I cannot import in python 3.5 because pyqt4 from the repo is built for python3.4. That's solved at least.

[zafields]

Just reporting here to add visibility, this "problem" also affects the GoogleCloudPlatform pip packages.

Instead of installing pip via apt-get you can manually download and install from a binary, which allows it to use the site-packages folder instead of the dist-packages.

Here is a very through issue resolution that outlines the solution required, but also show how much of point of confusion this really is.

[erindru]

Id just like to report another use-case - and that is installing packages that dont exist in pip.
For example, the MariaDB AX Bulk Data SDK (https://mariadb.com/downloads/mariadb-ax/data-adapters)
It is distributed via a .deb package only and cant be installed from pip even if you want to

[tianon]

For cases like those, our compiled Python isn't likely to be sufficient for
the package's needs, and it will likely make assumptions about the way
Debian/Ubuntu distribute and compile Python, so in those cases you're
definitely better off installing Python from Debian or Ubuntu directly to
then install their .deb file on top.

[tianon]

Closing, since this is resolved as much as it can be -- if you need packages from Debian, you should probably go FROM debian:xxx and install Python itself from Debian too (for the reasons noted above). :+1:

[rocketraman]

I'm not sure I'm following for what's gained by supporting this path besides encouraging people to mix Debian Python packages with pip install packages? (which is generally considered harmful)

Funnily enough, using the python image, I had issues installing scikit-learn, scipy, and numpy via pip, which let me to try installing dependencies using apt-get, which led me to this issue when that didn't work either.

After switching to FROM debian:buster the apt-get mechanism worked fine, but just for shits and giggles I decided to try the original pip3-based install on the debian image, and lo and behold, that worked fine too.

Long story short: the python:buster image didn't work for either the pip or apt-get mechanism of installing dependencies: the Debian image worked for both. This image wasted several hours of my time, for no apparently good reason.