Poetry: Poetry silently fails to save basic-auth creds, leading to failure to install private deps

[tsiq-oliver]

• [ ] I am on the latest Poetry version.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.

• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: python:3.7 base Docker image

• Poetry version: 1.0.0b5
• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/tsiq-oliver/bb36a0c705459ab85197ec71de580a24

Issue

When install-ing a dep from a private repo with basic auth, Poetry non-deterministically fails (you can assume the env vars in the first line are set appropriately):

poetry config http-basic.my-private-repo "${REPO_USERNAME}" "${REPO_PASSWORD}"
poetry install --no-dev --no-interaction -vvv
Skipping virtualenv creation, as specified in config file.
Installing dependencies from lock file

Package operations: XXX installs, XXX updates, 0 removals, XXX skipped

  [...]
  - Installing myprivatepackage (0.0.6081)

[TypeError]
quote_from_bytes() expected bytes

Traceback (most recent call last):
  File "/root/.poetry/lib/poetry/_vendor/py3.7/clikit/console_application.py", line 131, in run
    status_code = command.handle(parsed_args, io)
  File "/root/.poetry/lib/poetry/_vendor/py3.7/clikit/api/command/command.py", line 120, in handle
    status_code = self._do_handle(args, io)
  File "/root/.poetry/lib/poetry/_vendor/py3.7/clikit/api/command/command.py", line 171, in _do_handle
    return getattr(handler, handler_method)(args, io, self)
  File "/root/.poetry/lib/poetry/_vendor/py3.7/cleo/commands/command.py", line 92, in wrap_handle
    return self.handle()
  File "/root/.poetry/lib/poetry/console/commands/install.py", line 63, in handle
    return_code = installer.run()
  File "/root/.poetry/lib/poetry/installation/installer.py", line 74, in run
    self._do_install(local_repo)
  File "/root/.poetry/lib/poetry/installation/installer.py", line 286, in _do_install
    self._execute(op)
  File "/root/.poetry/lib/poetry/installation/installer.py", line 302, in _execute
    getattr(self, '_execute_{}'.format(method))(operation)
  File "/root/.poetry/lib/poetry/installation/installer.py", line 327, in _execute_install
    self._installer.install(operation.package)
  File "/root/.poetry/lib/poetry/installation/pip_installer.py", line 63, in install
    index_url = repository.authenticated_url
  File "/root/.poetry/lib/poetry/repositories/legacy_repository.py", line 224, in authenticated_url
    password=quote(self._auth.auth.password),
  File "/usr/local/lib/python3.7/urllib/parse.py", line 834, in quote
    return quote_from_bytes(string, safe)
  File "/usr/local/lib/python3.7/urllib/parse.py", line 859, in quote_from_bytes
    raise TypeError('quote_from_bytes() expected bytes')

Having looked through the code, we identified a couple of issues that combine to cause this.

1. Due to this keyring issue, poetry config can non-deterministically silently fail to save the password at all.

   Inside a container, there are only two backends - fail and chainer. If fail is chosen then keyring fails loudly with a RuntimeError, and poetry falls back to writing to its own config. However if chainer is chosen then keyring fails silently, and the password is saved nowhere.

2. poetry silently swallows keyring issues when trying to retrieve the password (no matter which backend keyring non-deterministically selects).

   This means that None is propagated all the way up to LegacyRepository, which then unconditionally attempts to pass it intourllib.quote which raises the exception in the trace above.

Suggested mitigations

1. Use the workaround described in the keyring issue - install keyrings.alt.

2. Fail loudly on poetry config if the password is not actually set - i.e. fail as early as possible in the workfow.

3. Fail loudly on password retrieval if keyring returns None - this is definitely (?) an error situation.

4. Give the user more control over where they expect the password to be written. (The silent fallback from keyring to plain-text file would already be a security risk in some scenarios, so may need re-designing.)

[stevegore]

As a workaround, I have this in some CI scripts:

    poetry config http-basic."Internal PyPI" $(cat ~/.secrets/PYPI_INTERNAL_USER) $(cat ~/.secrets/PYPI_INTERNAL_PASSWORD)
    if grep -q password ~/.config/pypoetry/auth.toml; then
        echo "Authentication was successful"
    else
        echo "Additional processing was required to authenticate"
        echo "password = \"$(cat ~/.secrets/PYPI_INTERNAL_PASSWORD)\"" >> ~/.config/pypoetry/auth.toml  # Resolve bug with preview version
    fi

It then gets removed after poetry install

[areeh]

To add to the suggestions, please also fail loudly if credentials are wrong for a private repository.

Current pre-release 1.0.0b8 appears to fail silently if it does not have correct creds for a private repo, so poetry instead fails with a solver error when the package is expected to be in the private repo instead of PyPi. You can imagine this combined with the silent failures mentioned in the issue make for a frustrating debugging dive.

Since private repos have been and still are quite error prone I'd appreciate at least loud errors so I can find workarounds.

[rizerzero]

Experienced the same bug, I fixed the poetry version which fixed the issue 👍 :✨

• pip install -Iv poetry==0.12.17

[andrmueller]

Experienced the same bug with python3 with docker.

ENV PYTHONIOENCODING=utf-8

at the beginning of my Dockerfile solved the issue.

[tomzx]

Could you guys please test with poetry>=1.0.1 (either 1.0.1 or 1.0.2). On my side, this issue seems fixed.

[bayoumi17m]

I used poetry == 1.0.2 and I do not have this problem anymore. It seems to have been resolved.

[finswimmer]

Thanks a lot for reporting this as solved. :+1: So I will close it.

[Qu4tro]

I'm getting the same error with Poetry 1.0.5

[TypeError]
quote_from_bytes() expected bytes

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/clikit/console_application.py", line 131, in run
    status_code = command.handle(parsed_args, io)
  File "/usr/local/lib/python3.8/site-packages/clikit/api/command/command.py", line 120, in handle
    status_code = self._do_handle(args, io)
  File "/usr/local/lib/python3.8/site-packages/clikit/api/command/command.py", line 171, in _do_handle
    return getattr(handler, handler_method)(args, io, self)
  File "/usr/local/lib/python3.8/site-packages/cleo/commands/command.py", line 92, in wrap_handle
    return self.handle()
  File "/usr/local/lib/python3.8/site-packages/poetry/console/commands/install.py", line 63, in handle
    return_code = installer.run()
  File "/usr/local/lib/python3.8/site-packages/poetry/installation/installer.py", line 74, in run
    self._do_install(local_repo)
  File "/usr/local/lib/python3.8/site-packages/poetry/installation/installer.py", line 286, in _do_install
    self._execute(op)
  File "/usr/local/lib/python3.8/site-packages/poetry/installation/installer.py", line 302, in _execute
    getattr(self, "_execute_{}".format(method))(operation)
  File "/usr/local/lib/python3.8/site-packages/poetry/installation/installer.py", line 327, in _execute_install
    self._installer.install(operation.package)
  File "/usr/local/lib/python3.8/site-packages/poetry/installation/pip_installer.py", line 63, in install
    index_url = repository.authenticated_url
  File "/usr/local/lib/python3.8/site-packages/poetry/repositories/legacy_repository.py", line 224, in authenticated_url
    password=quote(self._auth.auth.password),
  File "/usr/local/lib/python3.8/urllib/parse.py", line 839, in quote
    return quote_from_bytes(string, safe)
  File "/usr/local/lib/python3.8/urllib/parse.py", line 864, in quote_from_bytes
    raise TypeError("quote_from_bytes() expected bytes")

https://github.com/python-poetry/poetry/issues/1600#issuecomment-555310610
This workaround still works.