Pip: Requested (package) has different version in metadata

[potiuk]

Describe the bug

In Airflow, we are experiencing problems when we are trying to use the PIP released 2 days ago which has the new resolver on by default.

ERROR: Requested oauthlib[signedtoken]>=1.0.0 from https://files.pythonhosted.org/packages/e5/54/4f96c51b171cf3a64a04b8c5167268803205bc5943b5cdf70bd770727b88/oauthlib-1.1.0-1.tar.gz#sha256=0f786c5573248a38efa86c48c59c0c93140ac836ab2a246aeefd8f9039e999ba (from jira->apache-airflow==1.10.13) has different version in metadata: '1.1.0'

How to reproduce

• Setup an empty virtualenv for Python 3.6
• Upgrade to latest pip: pip install --upgrade pip
• pip --version should return 20.3
• Run this command:

pip install "https://github.com/apache/airflow/archive/v1-10-test.tar.gz#egg=apache-airflow[all]" --constraint https://raw.githubusercontent.com/apache/airflow/constraints-1-10/constraints-3.6.txt

• Observe the output. It will keep on finding good dependencies until

Collecting oauthlib[signedtoken]>=1.0.0
  Using cached oauthlib-3.0.2-py2.py3-none-any.whl (143 kB)
  Using cached oauthlib-3.0.1-py2.py3-none-any.whl (142 kB)
  Using cached oauthlib-3.0.0-py2.py3-none-any.whl (142 kB)
  Using cached oauthlib-2.1.0-py2.py3-none-any.whl (121 kB)
  Using cached oauthlib-2.0.7-py2.py3-none-any.whl (124 kB)
  Using cached oauthlib-2.0.6.tar.gz (127 kB)
  Using cached oauthlib-2.0.5.tar.gz (129 kB)
  Using cached oauthlib-2.0.4.tar.gz (127 kB)
  Using cached oauthlib-2.0.3.tar.gz (127 kB)
  Using cached oauthlib-2.0.2.tar.gz (125 kB)
  Using cached oauthlib-2.0.1.tar.gz (122 kB)
  Using cached oauthlib-2.0.0.tar.gz (122 kB)
  Using cached oauthlib-1.1.2.tar.gz (111 kB)
  Using cached oauthlib-1.1.1.tar.gz (108 kB)
  Using cached oauthlib-1.1.0-1.tar.gz (106 kB)
ERROR: Requested oauthlib[signedtoken]>=1.0.0 from https://files.pythonhosted.org/packages/e5/54/4f96c51b171cf3a64a04b8c5167268803205bc5943b5cdf70bd770727b88/oauthlib-1.1.0-1.tar.gz#sha256=0f786c5573248a38efa86c48c59c0c93140ac836ab2a246aeefd8f9039e999ba (from jira->apache-airflow[all]) has different version in metadata: '1.1.0'
````
Apparently, metadata in published oauthlib 1.1.0-1 is wrong snd points to 1.1.0.



The same command with the legacy resolver works fine:

pip install --use-deprecated legacy-resolver "https://github.com/apache/airflow/archive/v1-10-test.tar.gz#egg=apache-airflow[all]" --constraint https://raw.githubusercontent.com/apache/airflow/constraints-1-10/constraints-3.6.txt
```

Expected behavior

I expect the resolver does not get broken by broken metadata.

I've opened similar issue oauthlib as I am not sure who can fix it: https://github.com/oauthlib/oauthlib/issues/744

[brainwane]

Hi, @potiuk and thank you for your bug report! I'm sorry you're having trouble right now. Thank you for sharing your report with us. (If you don't mind, please also tell us what could have happened differently so you could have tested and caught and reported this during the resolver beta period.)

As you noted: you need a workaround while the root cause of this problem is being fixed, so you can choose the old resolver behavior using the flag --use-deprecated=legacy-resolver. This will work until we release pip 21.0 (see Deprecation timeline).

I expect the resolver does not get broken by broken metadata.

I understand that this was the previous behavior, but pip's maintainers have decided to make pip stricter and more consistent in dealing with incompatible, inconsistent, and broken combinations of metadata. We are doing this to pave the way for a lot of deeply-needed bugfixes and features, such as warning the user when uninstalling a package that other packages depend on, adding an "upgrade-all" command to pip, properly respecting constraints, and more.

I'll defer to others on diagnosing precisely what needs to be fixed in order to resolve this. Thanks again.

[notatallshaw]

I feel like this is still a bug with the new pip resolver. Reproducing it myself I get the same error (the previous line included):

INFO: pip is looking at multiple versions of oauthlib[signedtoken] to determine which version is compatible with other requirements. This could take a while.
Collecting oauthlib[signedtoken]>=1.0.0
  Downloading oauthlib-3.0.2-py2.py3-none-any.whl (143 kB)
  Downloading oauthlib-3.0.1-py2.py3-none-any.whl (142 kB)
  Downloading oauthlib-2.1.0-py2.py3-none-any.whl (121 kB)
  Downloading oauthlib-2.0.7-py2.py3-none-any.whl (124 kB)
  Downloading oauthlib-2.0.6.tar.gz (127 kB)
  Downloading oauthlib-2.0.5.tar.gz (129 kB)
  Downloading oauthlib-2.0.4.tar.gz (127 kB)
  Downloading oauthlib-2.0.3.tar.gz (127 kB)
  Downloading oauthlib-2.0.2.tar.gz (125 kB)
  Downloading oauthlib-2.0.1.tar.gz (122 kB)
  Downloading oauthlib-2.0.0.tar.gz (122 kB)
  Downloading oauthlib-1.1.2.tar.gz (111 kB)
  Downloading oauthlib-1.1.1.tar.gz (108 kB)
  Downloading oauthlib-1.1.0-1.tar.gz (106 kB)
ERROR: Requested oauthlib[signedtoken]>=1.0.0 from https://files.pythonhosted.org/packages/e5/54/4f96c51b171cf3a64a04b8c5167268803205bc5943b5cdf70bd770727b88/oauthlib-1.1.0-1.tar.gz#sha256=0f786c5573248a38efa86c48c59c0c93140ac836ab2a246aeefd8f9039e999ba (from jira->apache-airflow[all]) has different version in metadata: '1.1.0'

When searching for a version of a package that is compatible with other requirements Pip should not completely error out just because 1 version of that package had broken metadata. Pip should instead discard any version of the package that has broken metadata in the dependency search.

If no version with valid metadata can be found that's what Pip should be reporting, not that a very old version of the package had broken metadata.

[uranusjr]

Pip should not completely error out just because 1 version of that package had broken metadata. Pip should instead discard any version of the package that has broken metadata in the dependency search.

We complete agree.

https://github.com/pypa/pip/blob/30eeb9ceb10bf76316f091e8615a9c3d8025fff0/src/pip/_internal/resolution/resolvelib/candidates.py#L204-L214

There are, however, still may issues in pip that needed to be worked on, and this particular one is relatively low-impact and have alternative solutions. I understand it may feel urgent when it directly affects you, but there are really only a very small number of packages that have this issue (I believe this is the only one we know served on PyPI so far), and the package maintainer can resolve the issue by yanking the problematic release. The issue is therefore currently considered less urgent.

[potiuk]

Hi, @potiuk and thank you for your bug report! I'm sorry you're having trouble right now. Thank you for sharing your report with us. (If you don't mind, please also tell us what could have happened differently so you could have tested and caught and reported this during the resolver beta period.)

Very good question indeed. The problem in our case that for a long time we had conflicting requirements in Airflow (we have > 420 requirements in total, because Apache Airflow is an orchestrator and we have more than 70 different providers (GCP/AWS/Postgres etc.) - each with its own set of requirements. We are releasing Airflow 2.0 (Release candidate planned for next week) and we've worked hard to get the 2.0 out, and only last week I started to look at our deps to make them conflict-free. I managed to do it (and now we have automated system that keeps the dependencies automatically updated but also pip-checked - those depencies are going to be upgraded every time after they pass all tests and pip-check in our master build. We are using automatically generated set of "good" constraints : https://github.com/apache/airflow/tree/constraints-master - if you look at the history of those commits, you will see that they get automatically updated whenever pip install .[all] --upgrade --upgrade-strategy eager will produce a set fo constratins that:

• all tests pass with them
• the constraints pass the 'pip check` (this works as of last week - previously we had conflicts in those constraints)

We could not test the new resolver before we fixed the constraints, because we had those conflicting constraints. With those, the new resolver went into an endless loop :(. Which is of course totally understandable. Since I just fixed the constraints, that was really the first time we could run the new resolver. And it is quite a coincidence of schedule with the PIP 20.3 release and Airflow 2.0 release as we've already in the past also had some limitation w/regards of PIP for a while and we had to pin it iand update our documentation.

As you noted: you need a workaround while the root cause of this problem is being fixed, so you can choose the old resolver behavior using the flag --use-deprecated=legacy-resolver. This will work until we release pip 21.0 (see Deprecation timeline).

Thanks!. This is what we've done - we updated the documentation and told our users to use this for now (or downgrade to the previous version of PIP). In our official docker images, we pinned PIP to 20.2.4 for now. I would love to use the new resolver thought and wondered if there is a way to disable the strict check :). But i understand from the discussion that it is a design decision for now. I understand that there are more pressing issues, so we can go with that constraint for a while - not ideal and caused us a spike of issues and questions from our users already, so we would prefer if this is fixed of course, also maybe the oauthlib team will fix it (for example by removing the offending version?).

[potiuk]

Another input:

I looked how we can workaround it and I thought we can do it by limiting the oauthlib directly, but it seems we can't:

We already have this in our setup.py oauthlib!=2.0.3,!=2.0.4,!=2.0.5,<3.0.0,>=1.1.2 . It's a bit strange that in this resolver downloads "all versions" of the lib even if we have this limitation (>=1.1.2):

  Downloading oauthlib-2.0.1.tar.gz (122 kB)
  Downloading oauthlib-2.0.0.tar.gz (122 kB)
  Downloading oauthlib-1.1.2.tar.gz (111 kB)
  Downloading oauthlib-1.1.1.tar.gz (108 kB)
  Downloading oauthlib-1.1.0-1.tar.gz (106 kB)
ERROR: Requested oauthlib[signedtoken]>=1.0.0 from https://files.pythonhosted.org/packages/e5/54/4f96c51b171cf3a64a04b8c5167268803205bc5943b5cdf70bd770727b88/oauthlib-1.1.0-1.tar.gz#sha256=0f786c5573248a38efa86c48c59c0c93140ac836ab2a246aeefd8f9039e999ba (from jira->apache-airflow[all]) has different version in metadata: '1.1.0'

[potiuk]

Another question. Would removing or yanking the 1.1.0-1 of oauthlib version help ? I proposed this in https://github.com/oauthlib/oauthlib/issues/744

[uranusjr]

Yes, I’d say yanking is the best solution for now.

[ashb]

Does this error (where a single bad dist causes the resolver to stop) not leave the python ecosystem open to a "left-pad attack" -- a disgruntled package author of something used by lots of transitive deps uploads a purposefully broken dist, :boom: half of pypi becomes uninstallable.

[uranusjr]

With any luck, we’ll be in time to release a solution before it happens. But even if not, I believe the PyPI terms of use give the admins power to remove things uploaded in bad faith. This is slightly different from the left-pad problem since removal is within a user’s rights and the admin cannot restore it without consent. (Disclaimer: IANAL and do not involve in maintaining PyPI.)

[notatallshaw]

but there are really only a very small number of packages that have this issue (I believe this is the only one we know served on PyPI so far),

This is reasonable why this particular issue is lower priority, and I totally understand the difficult situation pip is in here.

and the package maintainer can resolve the issue by yanking the problematic release. The issue is therefore currently considered less urgent.

This is a little problematic in principle, if there is a failure in the pip resolver that a 3rd party package maintainers must resolve for my package dependencies to be resolved via pip this leads to a concern in the stability of using pip to handle such dependency resolution.

IMO, there should be a way to specify package versions from the pip dependency resolver ever having to consider, just as one can specify package versions to not be installed. Particularly as the resolver needs to download and extract each package version to assess the dependency requirements, which can explode the number of downloads and extraction required when dealing with large dependency trees.

In fact it's odd to me that resolver doesn't use the install specification to proactively cull possible package versions, e.g. when oauthlib!=2.0.3 is specified in the initial package requirements the resolver still downloads and extracts this version of the package. Is this also a #TODO? Or is there some existing solution we are all missing here?

I can understand though that the number of packages with large dependency trees is small and therefore lower down on pips priority list. Looking forward to see the resolver improve though and thanks for all your hard effort in getting it to provide at least some guarantees 😺 .

[pfmoore]

IMO, there should be a way to specify package versions from the pip dependency resolver ever having to consider

There is. You can add a constraints file with a line mypackage != 1.0.0 and pip will not consider that specific version of the package.

Is this also a #TODO?

Pip considers requirements and dependencies one by one, removing specific cases as we discover reasons they are invalid. The order in which we check things is theoretically irrelevant, but in practice can have a significant effect on how much work we do. We're working on improving that (#9185) but it's balancing a number of heuristics, so it's never going to be perfect for everyone.

What we don't have is a way to say "pretend version X of package Y simply doesn't exist, anywhere, ever". Users can simulate this by running a local PyPI proxy that strips that version out, but that's a very heavy-handed solution. It would be possible to add such a feature to pip (in effect, a config file that tells pip's "finder" to discard specific package/version combinations), but we don't have such a thing at the moment. I don't honestly know how many people need something like this (we're still trying to triage and categorise the various issue reports in this area) so at the moment I have no idea how much priority such a feature would get, but it's a possibility.

[notatallshaw]

There is. You can add a constraints file with a line mypackage != 1.0.0 and pip will not consider that specific version of the package.

I think I'm misunderstand what you mean by this or it doesn't work, I tried adding the following to the constraints file (which already has oauthlib==3.1.0):

oauthlib!=1.1.0
oauthlib!=1.1.0-1

And I still get the error:

  Using cached oauthlib-1.1.0-1.tar.gz (106 kB)
ERROR: Requested oauthlib[signedtoken]>=1.0.0 from https://files.pythonhosted.org/packages/e5/54/4f96c51b171cf3a64a04b8c5167268803205bc5943b5cdf70bd770727b88/oauthlib-1.1.0-1.tar.gz#sha256=0f786c5573248a38efa86c48c59c0c93140ac836ab2a246aeefd8f9039e999ba (from jira->apache-airflow[all]) has different version in metadata: '1.1.0'

Pip considers requirements and dependencies one by one, removing specific cases as we discover reasons they are invalid. The order in which we check things is _theoretically_ irrelevant, but in practice can have a significant effect on how much work we do. We're working on improving that (#9185) but it's balancing a number of heuristics, so it's never going to be perfect for everyone.

Thanks! Yes and understood. I followed the improvements of the conda resolver that faced similar issues (and it was still difficult in a more controlled environment no less), I totally appreciate what a difficult problem it is and pips work here is great.

[potiuk]

There is. You can add a constraints file with a line mypackage != 1.0.0 and pip will not consider that specific version of the package.

@pfmoore I can confirm it does not work this way.

We already had 'oauthlib!=2.0.3,!=2.0.4,!=2.0.5,<3.0.0,>=1.1.2, in our setup.py. I even explicitly tried earlier today (hoping that we can workaround it) with both != 1.1.0 and != 1.1.0-1 and the installation broke the same way trying to download 1.1.0-1 in both cases. Seems that PIP downloads in this particular case ALL versions no matter what constraints it had.

Just one more comment -> the constraints we have are for sure GOOD. They were prepared using 'pip freeze` from a working installation of Airlfow and they pass ''pip check' of the pip 20.2.4 , so I assume they are fully consistent, and whatever PIP 20.2.4 installed, pip 20.3 should as well. It's also concerning why those packages are downloaded AT ALL. From how I understand the new resolver works, the --constraints flag works differently than in pip 20.2 - it basically limits the version "visible" to the resolver. So the resolver should not even know anything about oauhtlib different than 3.0.1, because the constraint file of ours:

https://raw.githubusercontent.com/apache/airflow/constraints-1-10/constraints-3.6.txt

contains:

oauthlib==3.1.0

This is at least what I understood from your comment here: https://github.com/pypa/pip/issues/8210#issuecomment-628453750

This makes me a little worried about soundness of the design and stability of the new pip in general.

Why those different versions are downloaded in the first place?

[potiuk]

The maintainers of oauthlib were so kind that they yanked the offending release. @pfmoore, unfortunately it did not help.

I see the release yanked, but still have the same error :(

[terrdavis]

fyi, i encountered the same issue using a local simple index (so the package may be very out of date).

ERROR: Requested pypyodbc from file://[...]/packages/simple/pypyodbc/pypyodbc-1.3.5.2.zip has different version in metadata: '1.3.4'

For now, I'll just pin pip to 20.2.

[uranusjr]

I think the != clause needs to be added as a top-level requirement, or in a constraint file instead, otherwise pip may try to resolve other packages first (and fail), because the != specifier hasn’t been seen yet when it hits the invalid version.