Poetry: Cannot download packages from private PyPi repository using HTTP basic auth with Poetry 1.1.0 + old v1.0.x PypiCloud w/ default settings

[MasterNayru]

• [x] I am on the latest Poetry version.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Fedora 32
• Poetry version: Poetry 1.1.0
• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/MasterNayru/0014ec89ae05d8f7651d18d8a107ec90

Issue

We have been using Poetry to pull down packages from a private PyPi repository and everything has been working fine until Poetry 1.1.0. We are configuring poetry to talk to our private PyPi installation by HTTP basic auth, and that auth works perfectly fine to resolve which versions of a package to install. The problem seems to be that that same auth is then used in the requests to download wheels from PyPi, which causes the following error to occur:

$ poetry config http-basic.myprivaterepo <username> <password>
$ poetry update -vvv

<snip>

   2  ~/.pyenv/versions/3.8.5/lib/python3.8/site-packages/poetry/repositories/pypi_repository.py:454 in _download
       452│ 
       453│     def _download(self, url, dest):  # type: (str, str) -> None
     → 454│         return download_file(url, dest, session=self.session)
       455│ 
       456│     def _log(self, msg, level="info"):

   1  ~/.pyenv/versions/3.8.5/lib/python3.8/site-packages/poetry/utils/helpers.py:98 in download_file
        96│ 
        97│     with get(url, stream=True) as response:
     →  98│         response.raise_for_status()
        99│ 
       100│         with open(dest, "wb") as f:

  HTTPError

  400 Client Error: Bad Request for url: https://deckard-pip.s3.amazonaws.com/1234/my_broken_dependency/my_broken_dependency-0.1.3-py3-none-any.whl?AWSAccessKeyId=<key>&Signature=kz30gf304b%2F%2F93pQeUSPrto5MiE%3D&x-amz-security-token=<token>&Expires=1601690152

  at ~/.pyenv/versions/3.8.5/lib/python3.8/site-packages/requests/models.py:941 in raise_for_status
      937│         elif 500 <= self.status_code < 600:
      938│             http_error_msg = u'%s Server Error: %s for url: %s' % (self.status_code, reason, self.url)
      939│ 
      940│         if http_error_msg:
    → 941│             raise HTTPError(http_error_msg, response=self)
      942│ 
      943│     def close(self):
      944│         
      945│         called the underlying ``raw`` object must not be accessed again.

If I change the following lines in the poetry code:

   2  ~/.pyenv/versions/3.8.5/lib/python3.8/site-packages/poetry/repositories/pypi_repository.py:454 in _download
       453│     def _download(self, url, dest):  # type: (str, str) -> None
     → 454│         return download_file(url, dest, session=self.session)

changes to:

   2  ~/.pyenv/versions/3.8.5/lib/python3.8/site-packages/poetry/repositories/pypi_repository.py:454 in _download
       453│     def _download(self, url, dest):  # type: (str, str) -> None
     → 454│         return download_file(url, dest)

and re-run, everything works:

$ poetry update
Skipping virtualenv creation, as specified in config file.
Updating dependencies
Resolving dependencies... (41.8s)

No dependencies to install or update

It seems like the auth is needed to talk to the API for package version resolution but causes issues when it is also used for package downloads. If it makes any difference, I am using pypicloud as the backend for my private PyPi installation.
I am trying to be as brief as possible with my output as possible without dumping any keys or stuff like that. Please let me know if you need any more information or suggestions on what I should be changing in my configuration to get my stuff working again.

[abn]

@MasterNayru interesting. We recently identified that in 1.0.10 we did not apply authentication correctly for source specified in the pyproject toml. In typical circumstances we expect authentication to be used for both api queries as well as file downloads.

Am I correct in understanding that in your case; authentication is used to retrieve wheel direct links (ie. incl. tokens) but the expectation is that we do not send basic auth when downloading these wheels? If so, this is a bit tricky, as this would mean there are 2 use cases that are not necessarily compatible with each other.

[MasterNayru]

@abn I am expecting that if I am trying to download packages from S3 that, since pypicloud returns a URL with the necessary auth parameters in the download URL from the API requests, and since S3 seems to error out when the username/password auth parameters are provided, that they will somehow not be used as part of the requests for the downloads, which seemed to fit in line pretty well with the behaviour in the older versions.

[MasterNayru]

@abn I am going to close this issue as it would appear that pypicloud have, since I last looked at it, defaulted to the behaviour which poetry is enforcing with regards to auth for package downloads. Rather than try to support the old behaviour which pip works with, I will update my pypicloud installation and make use of the new behaviour. Cheers for bearing with us.

[pepastach]

We hit exactly the same problem with poetry 1.1.1 and pypicloud 1.0.10.
The way I understand it, this can't be solved by pypicloud upgrade. Here's my reasoning:

1. poetry makes a call to pypicloud
2. pypicloud returns a pre-signed URL pointing to our S3 bucket (the URL already contains AWS access key and token)
3. poetry makes a GET request (using requests library) but since it passes the session as @MasterNayru described in the issue, requests adds the Authorization header. This makes the request invalid. We verified it manually using curl.

Since it's poetry/requests who adds the authorization header, I don't see how this can be fixed on pypicloud side.

Please correct me (or reopen the issue 😉 ).

[pepastach]

pypicloud 1.0.11 introduced the ability to stream files through pypicloud. By briefly looking at the diff, it seems like we can configure pypicloud with pypi.stream_files = True. pypicloud should then return the package file directly instead of redirecting to S3.

We'll try pypicloud bump and report back.

[Katafalkas]

Hi. Ran into the same issue.
pypicloud==1.1.5
poetry==1.1.3

Tried both with and without pypi.stream_files = True - same issue.

Error is because of the headers being sent. The same url is downloadable using curl.

b'<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>InvalidArgument</Code><Message>Only one auth mechanism allowed; only the X-Amz-Algorithm query parameter, Signature query string parameter or the Authorization header should be specified</Message><ArgumentName>Authorization</ArgumentName><ArgumentValue>Basic XxXx...</ArgumentValue><RequestId>58D6A864A3D27683</RequestId><HostId>FpXxx...</HostId></Error>'

[Katafalkas]

@MasterNayru I would suggest that this issue should be considered as poetry issue. The reason being - same pypicloud server is working just fine with pip, and does not with poetry.

We run number of repositories and python packages, only one of them is with poetry and currently it does not work. We should either wait or help with fixing poetry or migrate to pip.

[abn]

@Katafalkas one aspect to note here is that pip and poetry have different uses for these URLs. One such case is that poetry stores these URLs in the lock file. Considering that the tokens used for these URLs are short-lived, it is not ideal to be used within a lock file. As far as I can tell the use of short-lived authorised URLs are not a defined behaviour. The use case in pip most likely works because search and retrival are treated seperately, this in-effect might allow this to work as expected.

Considering that pypicloud is a common component these days, we might need to look at how we can better support it. On the other hand however, PEP 503 does not define a mechanism for independent authentication for the file URLs. Typically, the authentication for the host domain is re-used.

Out of curiosity, are the domains different for the index and the file?

[Qu4tro]

They are for us (also using pypicloud), as the files are hosted on amazonaws.com and the index is on our domain.

[Katafalkas]

@Katafalkas one aspect to note here is that pip and poetry have different uses for these URLs. One such case is that poetry stores these URLs in the lock file. Considering that the tokens used for these URLs are short-lived, it is not ideal to be used within a lock file. As far as I can tell the use of short-lived authorised URLs are not a defined behaviour. The use case in pip most likely works because search and retrival are treated seperately, this in-effect might allow this to work as expected.

Considering that pypicloud is a common component these days, we might need to look at how we can better support it. On the other hand however, PEP 503 does not define a mechanism for independent authentication for the file URLs. Typically, the authentication for the host domain is re-used.

Out of curiosity, are the domains different for the index and the file?

The url of pypi-cloud server and the file served from S3 are different by default, but there is an option to passthrough url. Which makes both of those URLs the same.

[cereblanco]

This is how I setup private pypi

1. Edit pyproject.toml

[[tool.poetry.source]]
name = "myprivate_pypi"
url = "https://pypi.myprivate_pypi.com/simple/"

1. At terminal, add poetry config credentials for private_pypi
   poetry config http-basic.myprivate_pypi <username> <password>

2. Update lock with --no-update
   poetry lock --no-update

3. Add your library that is found at private pypi or poetry install
poetry add <my-packate-found-at-private>

@meanderingcode let me know if this one works for you

[MeanderingCode]

This is how I setup private pypi

1. Edit pyproject.toml

[[tool.poetry.source]]
name = "myprivate_pypi"
url = "https://pypi.myprivate_pypi.com/simple/"

1. At terminal, add poetry config credentials for private_pypi
   poetry config http-basic.myprivate_pypi <username> <password>

2. Update lock with --no-update
   poetry lock --no-update

3. Add your library that is found at private pypi or poetry install
poetry add <my-packate-found-at-private>

@meanderingcode let me know if this one works for you

@cereblanco Thank you. I discovered this yesterday when looking at changes related to legacy repositories. Edited my comment on your PR.