pip + twine installed: pip attempts to continuously create and use a "keyring"

[asottile]

Environment

• pip version: 20.0.2

$ pip freeze --all
bleach==3.1.4
certifi==2020.4.5.1
cffi==1.14.0
chardet==3.0.4
cryptography==2.9
docutils==0.16
idna==2.9
importlib-metadata==1.6.0
jeepney==0.4.3
keyring==21.2.0
pip==20.0.2
pkginfo==1.5.0.1
pycparser==2.20
Pygments==2.6.1
readme-renderer==25.0
requests==2.23.0
requests-toolbelt==0.9.1
SecretStorage==3.1.2
setuptools==46.1.3
six==1.14.0
tqdm==4.45.0
twine==3.1.1
urllib3==1.25.9
webencodings==0.5.1
wheel==0.34.2
zipp==3.1.0

$ python --version --version
Python 3.6.9 (default, Nov  7 2019, 10:44:02) 
[GCC 8.3.0]

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.1 LTS
Release:    18.04
Codename:   bionic

Description

Upon any pip install command, pip attempts to create a keyring (???) -- at first I thought I was being hacked 🤣 -- here's pip install babi

Expected behavior

I expect pip to not do that, there's no reason to create a keyring here when it's not going to be used.

How to Reproduce

1. pip install twine
2. pip install babi # can be any package, this is the one in the screenshot

Output

(see screenshot above)

I didn't want to create a keyring so I clicked cancel, it prompts again (12 times in total to install babi) and produces output like this:

$ pip install babi
WARNING: Keyring is skipped due to an exception: Failed to create the collection: Prompt dismissed..
WARNING: Keyring is skipped due to an exception: Failed to create the collection: Prompt dismissed..
Collecting babi
  WARNING: Keyring is skipped due to an exception: Failed to create the collection: Prompt dismissed..
  Using cached babi-0.0.7-py2.py3-none-any.whl (41 kB)
WARNING: Keyring is skipped due to an exception: Failed to create the collection: Prompt dismissed..
WARNING: Keyring is skipped due to an exception: Failed to create the collection: Prompt dismissed..
Collecting identify
  WARNING: Keyring is skipped due to an exception: Failed to create the collection: Prompt dismissed..
  Using cached identify-1.4.14-py2.py3-none-any.whl (97 kB)
WARNING: Keyring is skipped due to an exception: Failed to create the collection: Prompt dismissed..
WARNING: Keyring is skipped due to an exception: Failed to create the collection: Prompt dismissed..
Collecting babi-grammars
  WARNING: Keyring is skipped due to an exception: Failed to create the collection: Prompt dismissed..
  Using cached babi_grammars-0.0.13-py2.py3-none-any.whl (351 kB)
Requirement already satisfied: importlib-metadata>=1; python_version < "3.8" in ./venv/lib/python3.6/site-packages (from babi) (1.6.0)
WARNING: Keyring is skipped due to an exception: Failed to create the collection: Prompt dismissed..
WARNING: Keyring is skipped due to an exception: Failed to create the collection: Prompt dismissed..
Collecting onigurumacffi>=0.0.10
  WARNING: Keyring is skipped due to an exception: Failed to create the collection: Prompt dismissed..
  Using cached onigurumacffi-0.0.14-cp36-abi3-manylinux1_x86_64.whl (528 kB)
Requirement already satisfied: zipp>=0.5 in ./venv/lib/python3.6/site-packages (from importlib-metadata>=1; python_version < "3.8"->babi) (3.1.0)
Requirement already satisfied: cffi>=1 in ./venv/lib/python3.6/site-packages (from onigurumacffi>=0.0.10->babi) (1.14.0)
Requirement already satisfied: pycparser in ./venv/lib/python3.6/site-packages (from cffi>=1->onigurumacffi>=0.0.10->babi) (2.20)
Installing collected packages: identify, babi-grammars, onigurumacffi, babi
Successfully installed babi-0.0.7 babi-grammars-0.0.13 identify-1.4.14 onigurumacffi-0.0.14

[pradyunsg]

/cc @zooba

[gutsytechster]

I tried reproducing it but, I didn't find any issue. Is there some local keyring setting which is required for it to behave this way? I performed the following steps:

• pip install keyring
• pip install twine
• pip install babi

But all installations performed smoothly.

[asottile]

it's possible you have a keyring set up already? I encountered this on an ~essentially bare installation (as you can see from the screenshot I hadn't even taken the time to install guest additions and resize my VM window 😆)

[asottile]

I believe this is linux-specific as well

[asottile]

here's how I've been able to take a working machine and "break" it:

mv ~/.local/share/keyrings{,.old}

[gutsytechster]

I don't have any keyring setup by default and it's tested on a Linux machine(Linux Mint to be specific)

[asottile]

@gutsytechster do you have gnome-keyring or whatever installed (and probably needs to be running)? does ls ~/.local/share/keyrings list anything?

[gutsytechster]

It outputs

login.keyring  user.keystore

[asottile]

then yes, you have a keyring set up

[tykling]

I was hit by this today and dear god it is annoying, do you need any info from me

[pradyunsg]

We need someone familiar with keyring, to look at pip._internal.network.auth and see what would be a good way to resolve this issue. :)

[equaeghe]

This is a really bad bug. I first thought that I couldn't install anything any more. Then I discovered I could dismiss the pop-up dialog (which blocks all keyboard input for the whole desktop!). I had to dismiss it nine times for a package with two deps.

Version info:

$ pip --version
pip 19.3.1 from /usr/lib/python3.7/site-packages/pip (python 3.7)

I'm on Gentoo Linux with KDE Plasma. By default keyring uses KWallet and it works, I checked.

$ keyring --list-backends
keyring.backends.kwallet.DBusKeyring (priority: 5.1)
keyring.backends.SecretService.Keyring (priority: 5)
keyring.backends.chainer.ChainerBackend (priority: 10)
keyring.backends.fail.Keyring (priority: 0)

[a-jd]

Got the same bug after installing twine. Warning threw for pip install practically any package. The solution that worked for me was to just change my password using passwd. I am not sure what is happening in the backend but I gave it a shot after seeing https://askubuntu.com/questions/918712/the-login-keyring-did-not-get-unlocked-when-you-logged-into-your-computer.

[nikolas]

I'm seeing this issue with pip 20.1.1 on Fedora 31. Installing anything with pip now takes like 10 minutes.

[hroncok]

We need someone familiar with keyring, to look at pip._internal.network.auth and see what would be a good way to resolve this issue. :)

Let me see...

[hroncok]

Previously the logic was:

• if the response is 401, ask for user and password to supply

Now the logic is:

• before doing a request, check whether we have user+password for the URL it in the keyring
• if not, proceed without, if yes, use it
• if the response is 401, ask for user and password and optionally store it

I believe the proper algorithm should be:

• if the response is 401, check whether we have user+password for the URL it in the keyring
• if yes, use it, if not, ask for user and password and optionally store it

I am not familiar with keyring but I don't think that it is needed.

Looking further.

[hroncok]

I have this naïve patch:

diff --git a/src/pip/_internal/network/auth.py b/src/pip/_internal/network/auth.py
index ca729fcd..b19825e1 100644
--- a/src/pip/_internal/network/auth.py
+++ b/src/pip/_internal/network/auth.py
@@ -110,7 +110,7 @@ class MultiDomainBasicAuth(AuthBase):
         return None

     def _get_new_credentials(self, original_url, allow_netrc=True,
-                             allow_keyring=True):
+                             allow_keyring=False):
         # type: (str, bool, bool) -> AuthInfo
         """Find and return credentials for the specified URL."""
         # Split the credentials and netloc from the url.
@@ -250,8 +250,14 @@ class MultiDomainBasicAuth(AuthBase):

         parsed = urllib_parse.urlparse(resp.url)

+        # Query the keyring for credentials:
+        username, password = self._get_new_credentials(resp.url,
+                                                       allow_netrc=False,
+                                                       allow_keyring=True)
+
         # Prompt the user for a new username and password
-        username, password, save = self._prompt_for_password(parsed.netloc)
+        if not username or not password:
+            username, password, save = self._prompt_for_password(parsed.netloc)

         # Store the new username and password to use for future requests
         self._credentials_to_save = None

The idea is that keyring is only ever queried when handling a 401 response.

I have no idea how to properly test this with a password protected index. Do you have any?

---

This other patch can suppress any subsequent attempt to query the keyring when the first attempt failed:

diff --git a/src/pip/_internal/network/auth.py b/src/pip/_internal/network/auth.py
index b19825e1..588ca0d3 100644
--- a/src/pip/_internal/network/auth.py
+++ b/src/pip/_internal/network/auth.py
@@ -44,6 +44,7 @@ except Exception as exc:
 def get_keyring_auth(url, username):
     # type: (str, str) -> Optional[AuthInfo]
     """Return the tuple auth for a given url from keyring."""
+    global keyring
     if not url or not keyring:
         return None

@@ -69,6 +70,7 @@ def get_keyring_auth(url, username):
         logger.warning(
             "Keyring is skipped due to an exception: %s", str(exc),
         )
+        keyring = None
     return None

Which can reduce the most annoying aspect of this problem.

[hroncok]

How can I move this forward? I can submit a PR but I have no idea about how to effectively tests this (nor manually, nor automated).

[chrahunt]

There is a test here that could act as a starting example. Since I guess this would be a functional test, we could actually install a keyring lookalike in the environment of the pip under test or the real one + a plugin that acts like we dictate using e.g. environment variables or something more complicated.

[hroncok]

OK, I've opened https://github.com/pypa/pip/pull/8687 but I'd feel more confident if there was some index with user+password I could try manually as well.

[hroncok]

I'd feel more confident if there was some index with user+password I could try manually as well.

Verified.

[bfredl]

Why is pip _ever_ caring about anything that has with my keyring to do in the first place? When I have _never_ done any operation in the past that would make pip justified to even try access my keyring. I know python stuff can be messy and we all try our best to improve things, but this is surprising even by my standards of insanity.

[pradyunsg]

I'm gonna close this since #8744 fixed the bulk of this issue. #8687 is also open right now, but I think it's fine to track that separately. See also: #8719.

Thanks again @hroncok for the PR! ^>^

[asottile]

@pradyunsg I disagree with closing this, it still shouldn't prompt at all

EDIT: it also still prompts on every install so it _really_ isn't solved

[pradyunsg]

Reopened to track #8687 merging.

[qiujiangkun]

The same as I use twisted. I'm not sure whether they are the same.