Poetry: 1.1.4 regression custom repository does not work properly anymore.

[qiuwei]

• [x] I am on the latest Poetry version.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: MacOS 10.15.6
• Poetry version: 1.1.4
• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/qiuwei/69951b4284f51d28bbf2f7914336a0bb

Issue

We have a private custom PyPI repository set up using pywhaf.
The pywharf has a feature that it can redirect to an external index if a package is not found.
With latest version of poetry[1.1.4], this redirection seems not to work anymore.
Our primary PyPI is hosted at an internal url https://internal_url/simple
pip install works without any problem.

pip install --index-url https://internal_url/simple/ pandas

However, poetry fails to add the package. The log with -vvv:

Using virtualenv: /home/wqiu/.cache/pypoetry/virtualenvs/poetry-debug-ca-G0owh-py3.7
daozhi: 68 packages found for pandas *
PyPI: 68 packages found for pandas *
Using version ^1.1.3 for pandas

Updating dependencies
Resolving dependencies...
   1: fact: poetry-debug is 0.1.0
   1: derived: poetry-debug
   1: fact: poetry-debug depends on dzapi (^0.2.0)
   1: fact: poetry-debug depends on pandas (^1.1.3)
   1: fact: poetry-debug depends on pytest (^5.2)
   1: fact: poetry-debug depends on pytest (^5.2)
   1: selecting poetry-debug (0.1.0)
   1: derived: pytest (^5.2)
   1: derived: pandas (^1.1.3)
   1: derived: dzapi (^0.2.0)
daozhi: Downloading wheel: pytest-5.4.3-py3-none-any.whl
   1: Version solving took 0.321 seconds.
   1: Tried 1 solutions.

  Stack trace:

  25  ~/.cache/pypoetry/virtualenvs/playground-gu9jURa0-py3.7/lib/python3.7/site-packages/clikit/console_application.py:131 in run
       129│             parsed_args = resolved_command.args
       130│
     → 131│             status_code = command.handle(parsed_args, io)
       132│         except KeyboardInterrupt:
       133│             status_code = 1

  24  ~/.cache/pypoetry/virtualenvs/playground-gu9jURa0-py3.7/lib/python3.7/site-packages/clikit/api/command/command.py:120 in handle
       118│     def handle(self, args, io):  # type: (Args, IO) -> int
       119│         try:
     → 120│             status_code = self._do_handle(args, io)
       121│         except KeyboardInterrupt:
       122│             if io.is_debug():

  23  ~/.cache/pypoetry/virtualenvs/playground-gu9jURa0-py3.7/lib/python3.7/site-packages/clikit/api/command/command.py:171 in _do_handle
       169│         handler_method = self._config.handler_method
       170│
     → 171│         return getattr(handler, handler_method)(args, io, self)
       172│
       173│     def __repr__(self):  # type: () -> str

  22  ~/.cache/pypoetry/virtualenvs/playground-gu9jURa0-py3.7/lib/python3.7/site-packages/cleo/commands/command.py:92 in wrap_handle
        90│         self._command = command
        91│
     →  92│         return self.handle()
        93│
        94│     def handle(self):  # type: () -> Optional[int]

  21  ~/.cache/pypoetry/virtualenvs/playground-gu9jURa0-py3.7/lib/python3.7/site-packages/poetry/console/commands/add.py:174 in handle
       172│
       173│         try:
     → 174│             status = self._installer.run()
       175│         except Exception:
       176│             self.poetry.file.write(original_content)

  20  ~/.cache/pypoetry/virtualenvs/playground-gu9jURa0-py3.7/lib/python3.7/site-packages/poetry/installation/installer.py:103 in run
       101│         local_repo = Repository()
       102│
     → 103│         return self._do_install(local_repo)
       104│
       105│     def dry_run(self, dry_run=True):  # type: (bool) -> Installer

  19  ~/.cache/pypoetry/virtualenvs/playground-gu9jURa0-py3.7/lib/python3.7/site-packages/poetry/installation/installer.py:235 in _do_install
       233│             )
       234│
     → 235│             ops = solver.solve(use_latest=self._whitelist)
       236│         else:
       237│             self._io.write_line("Installing dependencies from lock file")

  18  ~/.cache/pypoetry/virtualenvs/playground-gu9jURa0-py3.7/lib/python3.7/site-packages/poetry/puzzle/solver.py:65 in solve
        63│         with self._provider.progress():
        64│             start = time.time()
     →  65│             packages, depths = self._solve(use_latest=use_latest)
        66│             end = time.time()
        67│

  17  ~/.cache/pypoetry/virtualenvs/playground-gu9jURa0-py3.7/lib/python3.7/site-packages/poetry/puzzle/solver.py:234 in _solve
       232│         try:
       233│             result = resolve_version(
     → 234│                 self._package, self._provider, locked=locked, use_latest=use_latest
       235│             )
       236│

  16  ~/.cache/pypoetry/virtualenvs/playground-gu9jURa0-py3.7/lib/python3.7/site-packages/poetry/mixology/__init__.py:7 in resolve_version
       5│     solver = VersionSolver(root, provider, locked=locked, use_latest=use_latest)
       6│
     → 7│     return solver.solve()
       8│

  15  ~/.cache/pypoetry/virtualenvs/playground-gu9jURa0-py3.7/lib/python3.7/site-packages/poetry/mixology/version_solver.py:84 in solve
        82│             while next is not None:
        83│                 self._propagate(next)
     →  84│                 next = self._choose_package_version()
        85│
        86│             return self._result()

  14  ~/.cache/pypoetry/virtualenvs/playground-gu9jURa0-py3.7/lib/python3.7/site-packages/poetry/mixology/version_solver.py:397 in _choose_package_version
       395│             version = locked
       396│
     → 397│         version = self._provider.complete_package(version)
       398│
       399│         conflict = False

  13  ~/.cache/pypoetry/virtualenvs/playground-gu9jURa0-py3.7/lib/python3.7/site-packages/poetry/puzzle/provider.py:437 in complete_package
       435│                     package.version.text,
       436│                     extras=list(package.dependency.extras),
     → 437│                     repository=package.dependency.source_name,
       438│                 ),
       439│             )

  12  ~/.cache/pypoetry/virtualenvs/playground-gu9jURa0-py3.7/lib/python3.7/site-packages/poetry/repositories/pool.py:135 in package
       133│             for idx, repo in enumerate(self._repositories):
       134│                 try:
     → 135│                     package = repo.package(name, version, extras=extras)
       136│                 except PackageNotFound:
       137│                     continue

  11  ~/.cache/pypoetry/virtualenvs/playground-gu9jURa0-py3.7/lib/python3.7/site-packages/poetry/repositories/legacy_repository.py:323 in package
       321│             return self._packages[index]
       322│         except ValueError:
     → 323│             package = super(LegacyRepository, self).package(name, version, extras)
       324│             package._source_type = "legacy"
       325│             package._source_url = self._url

  10  ~/.cache/pypoetry/virtualenvs/playground-gu9jURa0-py3.7/lib/python3.7/site-packages/poetry/repositories/pypi_repository.py:158 in package
       156│         extras=None,  # type: (Union[list, None])
       157│     ):  # type: (...) -> Package
     → 158│         return self.get_release_info(name, version).to_package(name=name, extras=extras)
       159│
       160│     def search(self, query):

   9  ~/.cache/pypoetry/virtualenvs/playground-gu9jURa0-py3.7/lib/python3.7/site-packages/poetry/repositories/pypi_repository.py:224 in get_release_info
       222│
       223│         cached = self._cache.remember_forever(
     → 224│             "{}:{}".format(name, version), lambda: self._get_release_info(name, version)
       225│         )
       226│

   8  ~/.cache/pypoetry/virtualenvs/playground-gu9jURa0-py3.7/lib/python3.7/site-packages/cachy/repository.py:174 in remember_forever
       172│             return val
       173│
     → 174│         val = value(callback)
       175│
       176│         self.forever(key, val)

   7  ~/.cache/pypoetry/virtualenvs/playground-gu9jURa0-py3.7/lib/python3.7/site-packages/cachy/helpers.py:6 in value
       4│ def value(val):
       5│     if callable(val):
     → 6│         return val()
       7│
       8│     return val

   6  ~/.cache/pypoetry/virtualenvs/playground-gu9jURa0-py3.7/lib/python3.7/site-packages/poetry/repositories/pypi_repository.py:224 in <lambda>
       222│
       223│         cached = self._cache.remember_forever(
     → 224│             "{}:{}".format(name, version), lambda: self._get_release_info(name, version)
       225│         )
       226│

   5  ~/.cache/pypoetry/virtualenvs/playground-gu9jURa0-py3.7/lib/python3.7/site-packages/poetry/repositories/legacy_repository.py:377 in _get_release_info
       375│         data.files = files
       376│
     → 377│         info = self._get_info_from_urls(urls)
       378│
       379│         data.summary = info.summary

   4  ~/.cache/pypoetry/virtualenvs/playground-gu9jURa0-py3.7/lib/python3.7/site-packages/poetry/repositories/pypi_repository.py:410 in _get_info_from_urls
       408│             # Prefer non platform specific wheels
       409│             if universal_python3_wheel:
     → 410│                 return self._get_info_from_wheel(universal_python3_wheel)
       411│
       412│             if universal_python2_wheel:

   3  ~/.cache/pypoetry/virtualenvs/playground-gu9jURa0-py3.7/lib/python3.7/site-packages/poetry/repositories/pypi_repository.py:431 in _get_info_from_wheel
       429│         with temporary_directory() as temp_dir:
       430│             filepath = Path(temp_dir) / filename
     → 431│             self._download(url, str(filepath))
       432│
       433│             return PackageInfo.from_wheel(filepath)

   2  ~/.cache/pypoetry/virtualenvs/playground-gu9jURa0-py3.7/lib/python3.7/site-packages/poetry/repositories/pypi_repository.py:450 in _download
       448│
       449│     def _download(self, url, dest):  # type: (str, str) -> None
     → 450│         return download_file(url, dest, session=self.session)
       451│
       452│     def _log(self, msg, level="info"):

   1  ~/.cache/pypoetry/virtualenvs/playground-gu9jURa0-py3.7/lib/python3.7/site-packages/poetry/utils/helpers.py:99 in download_file
        97│
        98│     with get(url, stream=True) as response:
     →  99│         response.raise_for_status()
       100│
       101│         with open(dest, "wb") as f:

  HTTPError

  404 Client Error: Not Found for url: https://internal_url/packages/9f/f3/0a83558da436a081344aa6c8b85ea5b5f05071214106036ce341b7769b0b/pytest-5.4.3-py3-none-any.whl#sha256=5c0db86b698e8f170ba4582a492248919255fcd4c79b1ee64ace34301fb589a1

  at ~/.cache/pypoetry/virtualenvs/playground-gu9jURa0-py3.7/lib/python3.7/site-packages/requests/models.py:941 in raise_for_status
      937│         elif 500 <= self.status_code < 600:
      938│             http_error_msg = u'%s Server Error: %s for url: %s' % (self.status_code, reason, self.url)
      939│
      940│         if http_error_msg:
    → 941│             raise HTTPError(http_error_msg, response=self)
      942│
      943│     def close(self):
      944│
      945│         called the underlying ``raw`` object must not be accessed again.

[sinoroc]

Is it related to this PR:

• _Use URL from response to support redirects (#3198)_

[jpyams]

It appears so. The description and output matches exactly the issue I saw when I created #3198.

[sinoroc]

@qiuwei Did you get a chance to try the fix proposed in https://github.com/python-poetry/poetry/pull/3198 ?