Pip: Keyring support should require an --enable-keyring flag

[pfmoore]

What's the problem this feature will solve?
Keyring support is enabled silently when the keyring module is installed in the user's system. There's no way for the user to say whether or not they want keyring support, or to turn it off when it causes an issue. The pip tests don't test the behaviour of the keyring module (just the integration) and there have been a number of issues reported which are down to keyring behaviour: #8634, #8485, #8443, #8090

Describe the solution you'd like
Add an --enable-keyring flag to pip that acts as an explicit "opt in" to keyring support. Users who want keyring enabled by default can set this via an environment variable or config file.

This would ensure that users have a clear understanding that they are opting into keyring usage, and would ensure that pip's behaviour can't be changed by unrelated changes to the user's system. It would also allow users having issues with keyring to easily disable it (by omitting the flag).

Also, if the user requests keyring support and the module is not available for some reason, it is easy to give an explicit message explaining the issue, whereas at the moment, pip just silently disables the feature with no explanation.

Alternative Solutions
Having an option to disable keyring support would allow users to opt out if they hit issues. But it would do nothing to help users to have a clear understanding of when they have keyring support enabled. It also doesn't address the "silent behaviour change" issue.

Additional context
When implemented, keyring support was actively debated, because the approach taken (enabling the feature if a particular package was available) is non-standard. Normally, pip vendors dependencies, and won't implement features that need modules that don't conform to its vendoring requirements. The keyring support was merged, but to be honest the concerns about the exception to our vendoring policy (see the thread starting here for details) were never really addressed. In my view, the current issues stem from that, and this issue is an attempt to contain the problem by making it explicitly "opt in".

[pradyunsg]

I'm strongly in favor of making the keyring support opt-in instead of opt-out.

Nearly all users I've interacted with have mentioned how the keyring support in pip is actually surprising rather than useful to them.

[chrahunt]

We generally see these kinds of requests alongside workflow-impacting changes:

1. this should be available in requirements.txt so that its usage can be standardized across a team or require fewer setup steps
2. a flag also needs to be present to disable it, so if it is enabled at a system config or user config level then it can be disabled on a per-query basis

I think it would be good to understand our position now enough to be able to help answer those questions later.

---

For the implementation of this, we would probably want to defer importing keyring at all until we confirm the opt-in. From discussions in at lease one keyring issue it's possible it could be made to do the backend check lazily, but there's no telling when that makes it into distributions. It would also be nice if we can say "just upgrade pip" to resolve issues like #8485.

[pfmoore]

Good points. Personally, I'd go with:

1. No, it's global only, and we explicitly don't support it in requirements.txt. That's to limit the complexity in general, and because there's no such support at the moment, so we shouldn't be adding yet more features as part of this issue, which is all about keeping the impact of keyring support limited.
2. Again, no. There's no way to disable it now short of uninstalling keyring or using whatever options keyring has to switch it off, why should we add one as part of this change?

For the implementation of this, we would probably want to defer importing keyring at all until we confirm the opt-in.

Probably, but only if it's easy to do. If it's complicated, then I'd say defer it to a separate feature request, and leave this as simply switching off the functionality.

(I should probably be clear here, I never actually liked the way the keyring feature was implemented, and if I had my way I'd just rip it out completely. But we're stuck with it now and my main aim is to limit the damage, and make it easier to not have it affect people who don't opt into it. I'm particularly offended by #8485 in this regard.)

Also, I should probably have added under Alternative Solutions - Implement a more robust plugin system for pip that allows reliable addition of "optional" features that rely on non-vendored libraries. But that's easy to say, and much harder to actually do, so I basically just consider keyring support as a demonstration of why such a thing is harder than it looks 🙂

[Julian]

Not sure if it's included in the issues alluded to here (or discussed in that original ticket that I haven't read yet), but similar issues to #6668 arise with the way keyring support works today as well.

Specifically -- because it's not vendored, if you're going to use e.g. tox to run some tests, and need keyring in that venv, it is difficult or impossible today to ensure keyring is installed at the right moment (so fetch keyring from PyPI, then proceed with tox installing everything else from potentially some other place that now needs keyring present).

[pradyunsg]

PRs to implement this are very welcome.

[Jmennius]

Is there a chance the keyring support will be improved by vendoring the package?
My concern is mostly because of virtual environments - they break the experience as the system keyring package is not accessible inside a venv.

[pfmoore]

The support cannot be vendored because it includes a C extension somewhere in the dependency chain, and pip only ships as a "pure Python" universal wheel.

[Jmennius]

The support cannot be vendored because it includes a C extension somewhere in the dependency chain, and pip only ships as a "pure Python" universal wheel.

Understood, otherwise, you would consider vendoring it?

[pfmoore]

Personally, I'd be against doing so. The known issues with keyring, with long delays on import, people getting advised to disable it, etc, suggest that vendoring it would add an unacceptable amount to our support burden. Also, I don't know how keyring backends work - we wouldn't want to vendor those so how much would vendoring just keyring help?

[Jmennius]

I agree that making it opt-in is likely the best solution.
But when you actually want to use it there is that barrier that keyring is actually a python package and is subject to related restictions (mostly because of venvs).

Usually, keyring backends work as daemons and are preinstalled on desktop operating systems. On Linux, communication with the daemon (Freedesktop SecretService API) is via DBus.
DBus may be the reason there are long import delays (there is an option to start the service on the first request to it), and 25 seconds resembles some timeout. I will check upstream if I can help.
Maybe, this is the reason people are having issues on Ubuntu WSL, maybe the daemon is just not installed (kind of headless installation, there is even a manual on how to install and use it in headless mode).

For example, with git, you have an ability to configure credential helpers, one of them is libsecret.
All you have to do to opt-in is install a package (git-credential-libsecret on Fedora) or compile a single file (Ubuntu/Debian somehow don't package it but the source is included...) and then configure it in your gitconfig.

As i understand from your previous comment the best alternative to bundling is to have a plugin system?

[pradyunsg]

@Jmennius That's how keyrings generally work -- not the keyring package in Python.

We can't vendor keyring since adapters/plugins for it assume that the package is available under the keyring namespace, and if we vendor it, it won't be -- it'd be pip._vendor.keyring.

[pfmoore]

@Jmennius jaraco/keyring#403 is probably where you would find the best place to try to improve things.

Honestly, pip isn't set up for this type of thing - yes, a plugin system would be the best approach, but we don't have one because we don't have a stable programming API for plugins to use, etc, etc. It's an ongoing debate that we aren't likely to resolve in the short term

The way keyring is handled is essentially a "better than nothing" solution, but like any such compromise it's not ideal for anyone 🙁

[Jmennius]

Thank you for your inputs, this clears up the situation for me at least.