Pip: ContextualVersionConflict on requirement to re-install when no egg_info and --process-dependency-links

[sublee]

• Pip version: 9.0.1
• Python version: 2.7.12
• Operating system: Ubuntu 16.04

Description:

In a particular case, unexpected ContextualVersionConflict occurs.

$ pip install --process-dependency-links -e .
Obtaining file:///.../pip-issue-4537
Exception:
Traceback (most recent call last):
  ...
ContextualVersionConflict: (simplejson 3.9.0 (.../env/lib/python2.7/site-packages), Requirement.parse('simplejson==3.10.0'), set(['pip-issue-4537']))

The conditions are:

• pip==9.0.1 (no problem in 8.1.2)
• The package to be installed has a dependency. The dependent package was already installed but different version. So it should be upgraded.
• Use --process-dependency-links option.
• Use -e option.
• No egg_info directory yet.

What I've run:

Here's the steps to reproduce this issue:

$ pip install pip==9.0.1
$ mkdir pip-issue-4537
$ cd pip-issue-4537
$ echo "from setuptools import setup" > setup.py
$ echo "setup(name='pip-issue-4537', install_requires=['simplejson==3.10.0'])" >> setup.py
$ pip install simplejson==3.9.0
$ pip install --process-dependency-links -e .
Obtaining file:///.../pip-issue-4537
Exception:
Traceback (most recent call last):
  ...
ContextualVersionConflict: (simplejson 3.9.0 (.../env/lib/python2.7/site-packages), Requirement.parse('simplejson==3.10.0'), set(['pip-issue-4537']))

But if we have a egg_info directory, this error doesn't occur:

$ python setup.py install_egg_info
$ pip install --process-dependency-links -e .
Obtaining file:///.../pip-issue-4537
Exception:
Traceback (most recent call last):
  ...
ContextualVersionConflict: (simplejson 3.9.0 (.../env/lib/python2.7/site-packages), Requirement.parse('simplejson==3.10.0'), set(['pip-issue-4537']))

[pradyunsg]

Thanks for filing this issue! You say that this error doesn't occur if there's an egg_info directory, yet your output suggests otherwise...

None the less, I'm unable to reproduce this issue using the steps you've provided.

Obtaining file:///Users/pradyunsg/.venvwrap/venvs/tmp-210afdf64bc6e0/pip-issue-4537
  DEPRECATION: Dependency Links processing has been deprecated and will be removed in a future release.
Collecting simplejson==3.10.0 (from pip-issue-4537==0.0.0)
  Downloading simplejson-3.10.0.tar.gz (77kB)
    100% |████████████████████████████████| 81kB 192kB/s 
Building wheels for collected packages: simplejson
  Running setup.py bdist_wheel for simplejson ... done
  Stored in directory: /Users/pradyunsg/Library/Caches/pip/wheels/43/c5/ef/edcebbb19becffd2ba75bf219afdbb4ca85198b2d909f1b31b
Successfully built simplejson
Installing collected packages: simplejson, pip-issue-4537
  Found existing installation: simplejson 3.9.0
    Uninstalling simplejson-3.9.0:
      Successfully uninstalled simplejson-3.9.0
  Running setup.py develop for pip-issue-4537
Successfully installed pip-issue-4537 simplejson-3.10.0

I'll tag this as invalid, for now.

[sublee]

Really that example doesn't fail. Would you try the new example?

$ pip install pip==9.0.1
$ mkdir pip-issue-4537
$ cd pip-issue-4537
$ echo "from setuptools import setup" > setup.py
$ echo "setup(name='pip-issue-4537', install_requires=['simplejson==3.10.0'])" >> setup.py
$ pip install --process-dependency-links -e .
$ rm -rf *.egg-info
$ pip install simplejson==3.9.0
$ pip install -e .
Obtaining file:///.../pip-issue-4537
Exception:
Traceback (most recent call last):
  ...
ContextualVersionConflict: (simplejson 3.9.0 (.../env/lib/python2.7/site-packages), Requirement.parse('simplejson==3.10.0'), set(['pip-issue-4537']))

After the failure, if we call pip install -e . again, it succeeds.

[pradyunsg]

Thanks! This example does crash.

Output of running all these commands in a clean virtualenv:

Requirement already satisfied: pip==9.0.1 in ./lib/python3.6/site-packages
Obtaining file:///Users/pradyunsg/.venvwrap/venvs/tmp-459722adeec489/pip-issue-4537
  DEPRECATION: Dependency Links processing has been deprecated and will be removed in a future release.
Collecting simplejson==3.10.0 (from pip-issue-4537==0.0.0)
Installing collected packages: simplejson, pip-issue-4537
  Running setup.py develop for pip-issue-4537
Successfully installed pip-issue-4537 simplejson-3.10.0
Collecting simplejson==3.9.0
Installing collected packages: simplejson
  Found existing installation: simplejson 3.10.0
    Uninstalling simplejson-3.10.0:
      Successfully uninstalled simplejson-3.10.0
Successfully installed simplejson-3.9.0
Obtaining file:///Users/pradyunsg/.venvwrap/venvs/tmp-459722adeec489/pip-issue-4537
Exception:
Traceback (most recent call last):
  File "/Users/pradyunsg/.venvwrap/venvs/tmp-459722adeec489/lib/python3.6/site-packages/pip/req/req_install.py", line 1025, in check_if_exists
    self.satisfied_by = pkg_resources.get_distribution(str(no_marker))
  File "/Users/pradyunsg/.venvwrap/venvs/tmp-459722adeec489/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 558, in get_distribution
    dist = get_provider(dist)
  File "/Users/pradyunsg/.venvwrap/venvs/tmp-459722adeec489/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 432, in get_provider
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
  File "/Users/pradyunsg/.venvwrap/venvs/tmp-459722adeec489/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 968, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/Users/pradyunsg/.venvwrap/venvs/tmp-459722adeec489/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 859, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pip._vendor.pkg_resources.ContextualVersionConflict: (simplejson 3.9.0 (/Users/pradyunsg/.venvwrap/venvs/tmp-459722adeec489/lib/python3.6/site-packages), Requirement.parse('simplejson==3.10.0'), {'pip-issue-4537'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/pradyunsg/.venvwrap/venvs/tmp-459722adeec489/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/Users/pradyunsg/.venvwrap/venvs/tmp-459722adeec489/lib/python3.6/site-packages/pip/commands/install.py", line 335, in run
    wb.build(autobuilding=True)
  File "/Users/pradyunsg/.venvwrap/venvs/tmp-459722adeec489/lib/python3.6/site-packages/pip/wheel.py", line 749, in build
    self.requirement_set.prepare_files(self.finder)
  File "/Users/pradyunsg/.venvwrap/venvs/tmp-459722adeec489/lib/python3.6/site-packages/pip/req/req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/Users/pradyunsg/.venvwrap/venvs/tmp-459722adeec489/lib/python3.6/site-packages/pip/req/req_set.py", line 521, in _prepare_file
    req_to_install.check_if_exists()
  File "/Users/pradyunsg/.venvwrap/venvs/tmp-459722adeec489/lib/python3.6/site-packages/pip/req/req_install.py", line 1036, in check_if_exists
    self.req.name
  File "/Users/pradyunsg/.venvwrap/venvs/tmp-459722adeec489/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 558, in get_distribution
    dist = get_provider(dist)
  File "/Users/pradyunsg/.venvwrap/venvs/tmp-459722adeec489/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 432, in get_provider
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
  File "/Users/pradyunsg/.venvwrap/venvs/tmp-459722adeec489/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 968, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/Users/pradyunsg/.venvwrap/venvs/tmp-459722adeec489/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 859, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pip._vendor.pkg_resources.ContextualVersionConflict: (simplejson 3.9.0 (/Users/pradyunsg/.venvwrap/venvs/tmp-459722adeec489/lib/python3.6/site-packages), Requirement.parse('simplejson==3.10.0'), {'pip-issue-4537'})

I'm not sure what's expected when an editable installed package's egg-info is removed. @dstufft or @pfmoore would be in a much better position to comment. :)

[sublee]

I think if the second attempt succeeds, the first attempt should also succeed.

[lynncyrin]

This issue I'm having may have the same cause? https://stackoverflow.com/questions/45848936/pip-contextualversionconflict-when-pushing-new-versions-on-heroku

[artgoldberg]

I encounter this problem in production.

[pfmoore]

In my view, you shouldn't remove the egg-info directory for a package that's installed editable. That sounds to me like user error, in the same way that manually modifying site-packages (say, by deleting package metadata). It's unfortunate (to say the least!) that setuptools creates the metadata directory in the project working directory when creating editable installs, but that's not something under pip's control.

I'm not against pip producing some sort of warning or a better error message for this situation, but it's hard to see how to do that in a practical way. If someone were to offer a PR though, that would be fine.

[artgoldberg]

Details:
We encounter this on the python Docker image, on CircleCI, running
pip2 install -e .

[NotSqrt]

Hi,

In case someone else encounters this situation:

I faced this issue on Gitlab CI when reusing the same virtualenv between builds.
When the previous build had run pip install -e ., the corresponding *.egg-info/ is cleaned up at the start of the next build, which will result in ContextualVersionConflict exception when running pip install -e ..

By removing the *.egg-link file in the reused virtualenv, I can rerun pip install -e . without problems.

Thanks !

---

Edit: The most reliable way : pip remove the packages installed as -e on CI clean-up (after_script with Gitlab CI).

[mastak]

Hi,

We encounter this at Circleci.
As a workaround you can make egg-info before:

python setup.py install_egg_info
pip install -e .

[chdsbd]

I recently encountered this. This bug is especially a problem in CI where pip caching is used extensively.