Pip: Unexplainable ContextualVersionConflict

[1313e]

Not sure how I am going to describe this, as I cannot find any way to reproduce this on my own machine, but for some reason, on Travis CI for Python 2.7.15, pip fails to uninstall the already present numpy v1.16.0 and install the requested numpy<1.16.0,>=1.12.0.
The build job can be viewed here: https://travis-ci.com/1313e/PRISM/jobs/178432270

I know that when running on Travis CI on Linux, it automatically comes with NumPy 1.16.0, which is incompatible with what I want to test for Python 2.7.
Therefore, when installing all requirements, it should uninstall that version and install a version that satisfies the condition (which is v1.15.4).
Until today, it did this perfectly fine and it still does on my machine(s).
However, for some odd reason, it does not do that anymore and simply throws an ContextualVersionConflict.

The only thing I changed today is that I combined two requirements files into a single one.
One has a package requirement that simply requires NumPy (emcee), while the other has the version restrictions.
This does work perfectly fine on my machine(s), even in fresh environments, so I am a bit confused why this pops up all of a sudden.

I know this may seem a bit of a weird issue, as I cannot give any way or method to reproduce the error, but I hope somebody knows what the problem is here.

Edit: It seems that this problem affects both pip v18.1 and v19.0.2.

[pradyunsg]

Does this also occur with pip 19.0.2 and updated setuptools / wheel?

[1313e]

@pradyunsg Yup, it occurs with that as well: https://travis-ci.com/1313e/PRISM/jobs/178462602

[cjerdonek]

Never mind, I see from the Travis CI that this occurred with pip 18.1

[1313e]

@cjerdonek What do you mean?
In the second job output I showed, the same error was produced while using pip v19.0.2.

[cjerdonek]

What happened is that I couldn't tell from your original report without digging into the Travis CI details whether your issue was limited to pip 19.0 and up. But digging in I saw the first report was about 18.1. (It's always a good idea to include the affected pip versions in the issue report text itself.)

[1313e]

Alright, yeah, forgot about that.
It happens for both v18.1 and v19.0.2.

[uranusjr]

I did some digging, and this seems to happen only when you

• Use Travis’s default environment
• Have an existing installation of package X
• Perform an editable install of package Y requiring an incompatible X

This is the minimal reproducible case I came up with:

# setup.py
from setuptools import setup

setup(
    name='pip-conflict-travis',
    install_requires=['six>=1.0.0,<1.11.0'],
)

# Failing .travis.yml
language: python
python: '2.7'

install:
  - python -m pip install six==1.11.0
  - python -m pip install -e .

script:
  -

As mentioned previously, this is only reproducible on Travis. The problem also does not exist if you create your own virtual environment; the following works:

# Working .travis.yml
language: python
python: '2.7'

install:
  - virtualenv --python=$(which python2.7) venv
  - ./venv/bin/pip install six==1.11.0
  - ./venv/bin/pip install -e .

script:
  -

So I am inclined to think this is a configuration on Travis’s part, and there’s not much pip can do at the moment.

[uranusjr]

Forgot to mention—The same failure can be observed on Travis with Ubuntu Trusty and pip 9.0.1 (the default runtime), so it is highly unlikely to be related to any recent packaging changes.

[1313e]

@uranusjr Interesting.
I already had the feeling it was Travis that was causing this problem, but nevertheless, it still feels to me like pip has something to do with it as well.
After all, why would it fail to parse in the requirement properly?

Also, before, I did not have this problem.
The only difference is that now pip will check for both requirements at the same time, while before they were done separately (two pip install -r requirements.txt calls instead of one containing the other).

Forgot to mention—The same failure can be observed on Travis with Ubuntu Trusty and pip 9.0.1 (the default runtime), so it is highly unlikely to be related to any recent packaging changes.

I agree, but it might be an indication that there is a long-living problem somewhere.

[cjerdonek]

Some suggestions:

• What happens if you run pip freeze in the Travis environment both before and after the pip install six==1.11.0 line?
• What do you see when you enable more verbose logging?
• What directory is Travis installing packages into?
• Does this also affect 3.x versions on Travis?

[uranusjr]

pip does not fail to parse the requirement. The failure means “the "best" so far conflicts with a dependency” (quoting the comment where the exception is raised; I have no idea what that means).

I plan to continue digging and see if I can find what causes this.

[pfmoore]

You're using an Ubuntu distribution, I believe, and the system provided pip? If so, then could it be that the Ubuntu/Debian's patches to pip to are the issue here (in particular, I'd be concerned about their override to make --ignore-installed and --user the default)?

[uranusjr]

@pfmoore It is definitely not the pip provided by apt’s python-pip, and Travis doc says it’s a virtualenv. I don’t know whether and how it’s different from a virtualenv created manually though.

[uranusjr]

Update:

I realised that six==1.11.0 was installed by default (like numpy) in Travis’s virtualenv, so I switched to use scripttest instead. The results are the same.

• This also happens on Python 3.6, and likely other Python versions as well.
• Output of pip install -vvv. Not particularly helpful.
• Packages are installed into /home/travis/virtualenv/python$PY_XYZ/lib/python$PY_XY/site-packages, as expected. This is an isolated virtualenv chosen based on the specified Python version, according to Travis documentation.

Environment before installing scripttest:

$ python -m pip list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
attrs (17.3.0)
mock (2.0.0)
nose (1.3.7)
numpy (1.13.3)
pbr (3.1.1)
pip (9.0.1)
pluggy (0.6.0)
py (1.5.2)
pytest (3.3.0)
setuptools (38.2.4)
six (1.11.0)
wheel (0.30.0)

$ python -m pip freeze
attrs==17.3.0
mock==2.0.0
nose==1.3.7
numpy==1.13.3
pbr==3.1.1
pluggy==0.6.0
py==1.5.2
pytest==3.3.0
six==1.11.0

$ ls $(python -c 'from distutils.sysconfig import get_python_lib; print(get_python_lib())')
attr            pbr-3.1.1.dist-info pytest-3.3.0.dist-info
attrs-17.3.0.dist-info  pip         pytest.py
easy_install.py     pip-9.0.1.dist-info setuptools
mock            pkg_resources       setuptools-38.2.3.dist-info
mock-2.0.0.dist-info    pluggy          setuptools-38.2.4.dist-info
nose            pluggy-0.6.0.dist-info  six-1.11.0.dist-info
nose-1.3.7.dist-info    py          six.py
numpy           py-1.5.2.dist-info  wheel
numpy-1.13.3.dist-info  __pycache__     wheel-0.30.0.dist-info
pbr         _pytest

After installing scripttest (and before pip install -e .):

$ python -m pip list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
attrs (17.3.0)
mock (2.0.0)
nose (1.3.7)
numpy (1.13.3)
pbr (3.1.1)
pip (9.0.1)
pluggy (0.6.0)
py (1.5.2)
pytest (3.3.0)
scripttest (1.3)
setuptools (38.2.4)
six (1.11.0)
wheel (0.30.0)

$ python -m pip freeze
attrs==17.3.0
mock==2.0.0
nose==1.3.7
numpy==1.13.3
pbr==3.1.1
pluggy==0.6.0
py==1.5.2
pytest==3.3.0
scripttest==1.3
six==1.11.0

$ ls $(python -c 'from distutils.sysconfig import get_python_lib; print(get_python_lib())')
attr            pip         scripttest-1.3.dist-info
attrs-17.3.0.dist-info  pip-9.0.1.dist-info scripttest.py
easy_install.py     pkg_resources       setuptools
mock            pluggy          setuptools-38.2.3.dist-info
mock-2.0.0.dist-info    pluggy-0.6.0.dist-info  setuptools-38.2.4.dist-info
nose            py          six-1.11.0.dist-info
nose-1.3.7.dist-info    py-1.5.2.dist-info  six.py
numpy           __pycache__     wheel
numpy-1.13.3.dist-info  _pytest         wheel-0.30.0.dist-info
pbr         pytest-3.3.0.dist-info
pbr-3.1.1.dist-info pytest.py

Interestingly, there are two .dist-info database for setuptools, but that doesn’t seem to be related. (I tried deleting the stray dist-info directory; does not change the outcome.)

[cjerdonek]

What happens if you try to reproduce the issue locally (or in a fresh virtualenv) starting with those same packages installed?

[uranusjr]

@cjerdonek Can’t reproduce locally, installation ends successfully. This gave me an idea, however—I tried to reproduce this with a fresh virtualenv on Travis, and voila! It fails.

Next step: Try to build an environment more similar to Travis’s 🤔

[uranusjr]

I think I’ve found the problem. This happens when you try to build an egg-info with a conflicting requirement in the environment.

# setup.py
from setuptools import setup
setup(
    name='conflict-test',
    install_requires=['scripttest!=1.3'],
)

# IMPORTANT: Trigger egg-info rebuild.
rm -rf ./conflict_test.egg-info

# Build a fresh environment.
rm -rf ./venv
python3.6 -m venv venv
PYTHON="$PWD/venv/bin/python"

# Install an incompatible requirement.
$PYTHON -m pip install scripttest==1.3

# FAILURE!
$PYTHON -m pip install -e .

When running pip install, pip tries to check that the metadata is ready with InstallRequirement.check_if_exists(). For an sdist/editable (I think, please correct me if I’m wrong), this means it looks for the egg-info directory, and sets context whether the requirement is satisfied or has conflicts.

https://github.com/pypa/pip/blob/cba6083f92eb6137c623f8c99b36e6b150b41e66/src/pip/_internal/req/req_install.py#L376-L416

From what I understand, the large try-except block here follows this logic:

1. Try to find an existing, compatible distribution for the package to use.
2. If no distributions are found, good (we’ll install it).
3. If a distribution is found, but it is incompatible, record the offending distribution.

The problem is how 3. is implemented. The code assumes that if get_distribution("{package}=={version}") fails with VersionConflict, get_distribution("{package}") (dropping the specifiers) should return the correct result. But in the reported case, VersionConflict is raised not because the to-be-installed package conflicts, but one of its dependencies do. So the second get_distribution("{package}") still fails, resulting in a strange error.

I am not sure how this should be fixed. Maybe pkg_resources.get_distribution() should grow a flag so it can be used without checking conflicts in dependencies, or maybe should not use this function to get an installed distribution in the first place. Or maybe pip should uninstall that conflicting dependency (it will if this code block works anyway).

@1313e A workaround is to run python setup.py egg_info before pip install -e. This makes pip choose the 1. code path above, avoiding the bug.

[cjerdonek]

Great! And why was this limited to Travis? I'm assuming that was a red herring. Were you able to reproduce locally after all?

[cjerdonek]

Re: your proposed options, sometimes adding a flag to an existing function adds undesired complexity because of branching and making the call sites relying on that flag harder to find. Would it make sense to add a function to pkg_resources that is parallel to and shares code with get_distribution()? That way logic wouldn't be getting duplicated.

[uranusjr]

Yes I was! I wasn’t able to reproduce before because my test case was either too clean or not clean enough (a prebuilt egg-info resolves the problem). Travis has an environment right in the sweet spot, and local reproduction is easy once I know where that spot is :p

[1313e]

@uranusjr Wow, that turned out to be a lot harder and more complex than I thought.
I will use that workaround for now.

[Arusekk]

What is the status on this?

[pradyunsg]

@uranusjr This is a fairly big deviation from the resolvelib work for you, but any idea what we could change in pip to make this less likely for end users?

[1313e]

Does this not have to do with the dependency resolver issues discussed in #988?

[uranusjr]

Does this not have to do with the dependency resolver issues discussed in #988?

Yes, eventually the resolver work would make this whole block obsolete.

---

Any idea what we could change in pip to make this less likely for end users?

If we switch to importlib.metadata (#7413) it would stop being a problem for Python 3 users. The code here is not wrong in general, just using the wrong function (pkg_resources.get_distribution()).

[pradyunsg]

OK great! Given that our broader chunks of work would end up fixing this, I'll stop spending time trying to understand why we're having this issue; and instead spend it on progressing on those tasks. :)