Pip: Pandas fails to install on latest pre-release candidate

[davidjlloyd]

• Pip version: 10.0.0b2
• Python version: PyPy 5.10.0 with GCC 6.2.0
• Operating system: Ubuntu 16.04

Description:

We are creating a new virtualenv using PyPy and then attempting to install the package 'pandas'. Everything is running inside a clean Docker container. This bug does not occur with the same conditions and Pip 9.0.3. We are using a DevPI server to host our own packages, which mirrors content from PyPI. This does not appear to be an issue with Cython being available, as I can install it directly.

What I've run:

root@9b34418ce20d:/# /opt/pypy/bin/pypy /usr/lib/python2.7/dist-packages/virtualenv.py -p /opt/pypy/bin/pypy /opt/pypyenv
Already using interpreter /opt/pypy/bin/pypy
New pypy executable in /opt/pypyenv/bin/pypy
Installing setuptools, pkg_resources, pip, wheel...done.
root@9b34418ce20d:/# /opt/pypyenv/bin/pip freeze --all
cffi==1.11.2
greenlet==0.4.12
pip==10.0.0b2
pkg-resources==0.0.0
readline==6.2.4.1
setuptools==39.0.1
wheel==0.31.0

root@9b34418ce20d:/# /opt/pypyenv/bin/pip install pandas
Looking in indexes: https://devpi.myindex/root/myindex/+simple/
Collecting pandas
  Downloading https://devpi.myindex/root/pypi/+f/c7a/2757b60774825/pandas-0.22.0.tar.gz (11.3MB)
    100% |################################| 11.3MB 1.5MB/s 
  Could not find a version that satisfies the requirement Cython (from versions: )
No matching distribution found for Cython

root@9b34418ce20d:/# /opt/pypyenv/bin/pip install cython
Looking in indexes: https://devpi.myindex/root/myindex/+simple/
Collecting cython
  Downloading https://devpi.myindex/root/pypi/+f/c54/9effadb52d90b/Cython-0.28.1.tar.gz (1.8MB)
    100% |################################| 1.8MB 2.8MB/s 
Building wheels for collected packages: cython
  Running setup.py bdist_wheel for cython ... done
  Stored in directory: /root/.cache/pip/wheels/8e/21/58/4a0b5de660445a5a2c957575e2de6c2465df8c7c8310ae2e42
Successfully built cython
Installing collected packages: cython
  The scripts cygdb, cython and cythonize are installed in '/opt/pypyenv/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed cython-0.28.1

[benoit-pierre]

Does it work if you try: /opt/pypyenv/bin/pip install Cython (with a capital C)? Does it work if you don't use your own index?

[benoit-pierre]

Also, what is the output of /opt/pypyenv/bin/pip install -vv pandas?

[pradyunsg]

Ping @davidjlloyd!

[jfloff]

I am also experiencing the same problem. I'm within a Docker container (Alpine) and don't have the /opt/pypyenv/bin/pip executable.

[davidjlloyd]

I've recreated an almost minimal example for people to play with in Docker:

FROM ubuntu:16.04

RUN apt-get update && apt-get install -y bzip2 curl python-virtualenv
RUN curl -sS -L https://bitbucket.org/pypy/pypy/downloads/pypy2-v5.10.0-linux64.tar.bz2 | tar -C /opt -xjpf - && ln -s pypy2-v5.10.0-linux64 /opt/pypy
RUN virtualenv -p /opt/pypy/bin/pypy /opt/pypyenv
RUN /opt/pypyenv/bin/pip install pandas

This gives the reported output. You could probably recreate using CPython and installing Pandas using source distributions, but that currently hits another error first.

[pfmoore]

@davidjlloyd please can you provide the output with -vv as that should show what candidates were tried and discarded.

[davidjlloyd]

Here's the full output from building that Dockerfile with the additional -vv option for pip: output.txt

[pfmoore]

The key in there is probably "No sources permitted for Cython". I don't know why that's happening - is there some configuration (either in your setup or in Pandas) that would stop pip from installing Cython from source?

[pfmoore]

Pandas has Cython as a build requirement in its pyproject.toml. However, pip 10 requires that all build dependencies are available in binary form - this is a limitation of our initial PEP 518 support, documented here.

Until pip adds support for building build dependencies from source, you'll need to install Cython manually in your build environment (which will involve using the --no-build-isolation flag.

[davidjlloyd]

Whilst build isolation seems like a valuable feature, the fact that it requires binary distributions for all dependencies makes the default behaviour of enabling it a significant breaking change over Pip v9.0.3. It is especially bad for variant Python implementations like PyPy, which typically don't have any wheels available to them. Is there a strong reason to not default the behaviour to --no-build-isolation instead, at least until we can install build dependencies from source?

[davidjlloyd]

Running my example with the --no-build-isolation flag also fails with a different error: output.txt

Unfortunately this means I cannot verify that it works if we don't have build isolation enabled.

[pfmoore]

Your new error doesn't seem to be related to pip - "cc not found" looks more like a problem with your environment.

It's not build isolation that requires binary dependencies, it's pp's current implementation of PEP 518 support (pyproject.toml). And while pip's support for PEP 518 is incomplete at the moment, we've not made any secret of that fact. In the absence of any full implementations of PEP 518, it's arguable that Pandas' adoption of it for specifying a dependency like Cython which needs platform-specific wheels was premature.

[davidjlloyd]

Your new error doesn't seem to be related to pip - "cc not found" looks more like a problem with your environment.

Yup, that was my mistake. I was trying to put up any information I could in a hurry before leaving last night. After installing a few more dependencies it works fine, so at least we have a simple fix after Pip 10 is released.

I'll report this issue against Pandas as well in case they want to address it, but it sounds like we're just going to have to accept this Pip upgrade will have teething problems and deal with them when they happen. Obviously the ideal for us would be support for source distributions as part of the PEP 518 implementation in Pip, but I'm not holding my breath for this to happen in time for the next major release.

[cdagraca]

We are experiencing this on pip v10.0.0. We've had to pin to 9.x for docker containers using pandas for the time being as the builds suddenly broke over the weekend with the release.

[Davidnet]

I'm also encountering this error, I usually work with embedded devices, and providing binaries is not a plausible option. There should be a flag to pip to behave like pip 9

[pradyunsg]

@Davidnet --no-build-isolation

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.