Poetry: Poetry not respecting order of evaluation for repository sources

[TheFriendlyCoder]

• [X] I am on the latest Poetry version.

• [X] I have searched the issues of this repo and believe that this is not a duplicate.

• OS version and name: Mac OS 10.15.3

• Poetry version: 1.0.5

Issue

So, I am trying to configure a pyproject.toml file that supports pulling files from pypi.org whenever possible (ie: because on my site the performance is better) but will support pulling packages that aren't found on pypi.org from a secondary location (ie: a private pypi repo). So to start with I added a section like what follows to a sample toml file for testing:

[[tool.poetry.source]]
name = 'default'
url = 'https://pypi.python.org/simple'
default = true

[[tool.poetry.source]]
name = "private"
url = "https://private/server/url
secondary = true

[tool.poetry.dependencies]
python = "^3.6"
sphinx = "*"

Based on the docs, I was under the impression that by putting default=true on the first repo config, and secondary=true on the other one, that my goal would be achieved: if a package exists on pypi.org it'd pull it from there and if not it'd pull it from my secondary repo. However, that does not seem to be the case.

For the sake of this discussion I am using average performance metrics from running poetry lock on this toml file and comparing the length of time it takes to resolve the dependencies of the one Python package mentioned (ie: sphinx which is a standard Python package available on pypi.org).

So, running poetry lock on this file using the configuration I posted above takes about 20-25 seconds to complete. For comparison purposes I simply removed the second source definition from the toml file giving me:

[[tool.poetry.source]]
name = 'default'
url = 'https://pypi.python.org/simple'
default = true

[tool.poetry.dependencies]
python = "^3.6"
sphinx = "*"

Re-running the same common against this toml file takes between 2 and 3 seconds to complete - a full order of magnitude difference in performance. Now, for my 3rd and final test I removed the first repo source leaving just the second one, giving me the following configuration block:

[[tool.poetry.source]]
name = "private"
url = "https://private/server/url
secondary = true

[tool.poetry.dependencies]
python = "^3.6"
sphinx = "*"

Re-running the same command again takes about the same 20-25 seconds to complete (NOTE: our private pypi repository is also a mirror of the public pypi.org repo so I can compare the same package versions and resolution times in all cases).

So, based on these results, the only explanation I can see is that when the second repository definition is in place the public pypi.org repo is being ignored or something and the secondary / private repo is still being accessed for some reason.

My expectation here is that poetry would iterate over the various repos defined in the toml file in order of declaration. For each package listed in the toml file it should then try to access / index each package against each repository stopping when it finds the first match. If this were true then I would have expected my first test case above to perform identically to the second one because the package I'm testing (sphinx) should exist on pypi.org, and all of it's transitive dependencies will be available there as well. However this is obviously not the case.

So my question is - is this a bug or is there some additional configuration options I need to specify to get this to work as I was expecting?

[TheFriendlyCoder]

For kicks I just tried to run a few similar / comparable tests using pip directly (which poetry makes use of under the hood), making use of the index-url and extra-index-url parameters and I seem to be getting very similar results. Making the main index URL my private repo is slow, as is making my private repo an "extra" index and leaving the pypi.org as the default index URL, but when I use pypi.org exclusively the performance is better.

So maybe this is some odd limitation of pip and it just so happens that poetry is sharing that functionality.

I wonder if this is something new that has changed in a recent version of pip / poetry or something. I am pretty sure I've tried scenarios like this in the past with different results. I don't have detailed metrics from those days-gone-past though so I can't say for sure. :(

Still - any input anyone might have on whether there is anything that can be done to help mitigate this problem would be appreciated.

[TheFriendlyCoder]

NOTE: if it helps, I could create a map / list of packages that I know are exclusively only available on my private pypi server and thus could associate those packages with my mirror, and all others to the main pypi.org site. Not sure if that is possible or not though.

[madig]

I'm seeing the same (using a private repo as a secondary source) and am scratching my head at the following output:

> poetry update -vvv
Using virtualenv: C:\[...]\.venv
Updating dependencies
Resolving dependencies...
   1: fact: XXX is 0.39.5
   1: derived: XXX
   1: fact: XXX depends on pyparsing (^2.2)
[...]
   1: fact: XXX depends on rope (^0.17.0)
   1: selecting XXX (0.39.5)
   1: derived: rope (^0.17.0)
[...]
   1: derived: YYY (^1.32)
PyPI: No packages found for YYY >=1.32,<2.0
dama: https://PRIVATE_PYPI/simple/YYY/
dama: 52 packages found for YYY >=1.32,<2.0
PyPI: No release information found for rope-0.2, skipping
[...]]
PyPI: No release information found for rope-0.9, skipping
PyPI: No release information found for rope-0.9.1, skipping
PyPI: 1 packages found for rope >=0.17.0,<0.18.0
dama: https://PRIVATE_PYPI/simple/rope/
   1: Version solving took 1.902 seconds.
   1: Tried 1 solutions.

[Crash because our private PyPI does not forward to the official one using https]

The last lines seem to implicate that rope is found on the official PyPI, but then poetry goes off and queries the private repo for it anyway?!

[alimantu]

I face same issue here, it seems to be, that poetry/pip checks packages for both default and secondary sources. It makes sense if we are talking about robustness - you'll always get version that will fit you best from poetry/pip point of view, but even so I would suggest at least some key/argument to avoid this deep check in case package fit the dependencies already found during search in some previous repo.

[sinoroc]

Is that still an issue?
Was it maybe fixed by https://github.com/python-poetry/poetry/pull/3251 released in _version 1.1.4_?
Is that possibly fixed by the newest, unreleased, suggested solution in https://github.com/python-poetry/poetry/pull/3406?