Pip: Please add support for credentials via environment variables

[venthur]

• Pip version: all
• Python version: all
• Operating system: all

Description:

We're using pip in a CI/CD pipeline to install packages from a private repository protected by username/password. Currently there are two options to pass those credentials to pip, either encode it directly in the URL or create a pip.conf file. Both options are not very attractive. The first option would entail to have those credentials hard coded in the source code, the second one would mean we'd have to generate this config file during the build process.

Most CI/CD build pipelines support some kind of "secret variables", which is a fancy word for environment variables that you can set in the CI/CD and that will be enabled in the build pipeline. This is usually the way to pass secrets.

It would be very helpful if pip would also support some mechanism to read secrets from environment variables.

See also: https://www.jfrog.com/confluence/display/RTF/PyPI+Repositories#PyPIRepositories-UsingCredentials for a realistic use case.

[pradyunsg]

Adding the word authentication so that this shows up in my searches. :)

[xavfernandez]

Note that you can pass the --index-url option containing the login/password via the PIP_INDEX_URL environment variable.

[cmcginty]

Also wanted to point out that you can pass --extra-index-url with PIP_EXTRA_INDEX_URL and this will override the value set in requirements.txt.
So you can have a bare URL for source control and then set the environment value with secrets in your CI/CD.

[countergram]

I would not recommend setting PIP_INDEX_URL with a password unless you know you are using -q since otherwise the index url is logged including the password.

[pradyunsg]

I would not recommend setting PIP_INDEX_URL with a password unless you know you are using -q since otherwise the index url is logged including the password.

Not in pip 18.1 or later.

[jahickey]

Thank you for adding the expansion of environment variables in requirement files. However, I was wondering if environment variables could be implemented for Pip similar to Twine? With Twine (especially for CI) you just need to set TWINE_USERNAME and TWINE_PASSWORD as environment variables in the CI. Thus, there's no need to add the username and password to the repository URL's.

Just curious.

[pradyunsg]

There's keyring support that's integrated and up for the next release -- #5948.

[lhupfeldt]

I'm going to submit a PR to accept taking credentials from env variables. to make it much simpler to integrate safely with CI servers. Credentials should not be specified as command line options in any way as they may easily be leaked in logs or seen in process listings.

[ric79]

Hello, still open...

The PR was closed, even if PIP_PASSWORD was a nice idea.

[uranusjr]

Feel free to propose a new one if you think it is a good idea.

[ric79]

Sorry, I have not the knowledge to implement this PR. Is it possible to resubmit that PR.
It's quite strange having still now this strong security issue.

[TimOrme]

Wanted to chime in and voice support for this. It looks like the PR was closed, but maybe is still a viable option.

What would be the process for reviving it? Could someone else just open a new copy of the existing PR, or should we wait and see if @lhupfeldt can revive it?

[lhupfeldt]

You are welcome to reuse/reopen my PR. I just gave up originally because
there was some resistens from core developers.

On Wed, 22 Jul 2020, 17:54 Tim Orme, [e-mail] wrote:

Wanted to chime in and voice support for this. It looks like the PR was
closed, but maybe is still a viable option.

What would be the process for reviving it? Could someone else just open a
new copy of the existing PR, or should we wait and see if @lhupfeldt
https://github.com/lhupfeldt can revive it?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pypa/pip/issues/4789#issuecomment-662534589, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAMCZSIVUVXEOAXGYE62H6LR44DSPANCNFSM4D7OXPEQ
.

[ric79]

Hello, I think that the PR was fantastic and it works similar to twine. I do not like at all putting my credentials on the file system.
Please resubmit again. I have not knowledge about this kind of procedures

[pfmoore]

You'll hit the same resistance again, I suspect. This is precisely the sort of issue that keyring support was intended to avoid - pip needing to implement multiple mechanisms for handling authentication, each for a particular (entirely valid) use case. If keyring support (or keyring itself) isn't sufficient for this use case, we should be improving them, not implementing an alternative mechanism.

[ric79]

I do not find any article about how to use keyring for "Python3 pip + virtualenv". Could you point me a link?

[lancetarn]

If keyring support (or keyring itself) isn't sufficient for this use case, we should be improving them, not implementing an alternative mechanism.

This is a reasonable request, but it also doesn't seem like keyring is designed for the problem of CI/CD or automated builds or however you want to think of the issue that env var auth is trying to solve. The fact that it needs a "headless linux" section seems like an indicator of this. Maybe I'm wrong. Either way, this comment from the PR seems to capture my situation nicely:

If I understand correctly you expect me to:
1) Install the python keyring package on my systems (without using pip)
2) Implement my own keyring backend and install that on my systems (without
using pip)
3) Configure keyring to on all my systems to use my special backend.

That sounds like so much effort that I would rather munge my pip config/requirements.txt to inject credentials on the fly during builds.

If specifying explicit environment variables that pip will use for auth feels like a slippery slope, another option that seems to work for npm would be to attempt to replace things that look like environment variables in configuration files. This would at least allow one to set index-url with userinfo parts like https://${MY_USER}:${MY_PASS}@secretpypi.example.org and inject the relevant variables in CI. I don't know the pip code base at all or if that is feasible; just a thought that occurred to me. It also has the advantage of allowing the user to auth against multiple indexes if necessary by specifying different env vars for each.

[pfmoore]

it also doesn't seem like keyring is designed for the problem of CI/CD or automated builds or however you want to think of the issue that env var auth is trying to solve

Have you raised that with the keyring project? Honestly, that's all we're suggesting here, and we're getting a lot of pushback. Pip added support for keyring, in a good-faith attempt to handle the requests we were getting for a mechanism to store credentials outside of pip¹. We were led to understand that the submitted PR was a good solution to this issue, and we took it on trust that keyring did the job we'd been told that it did.

To date, no-one has demonstrated that keyring isn't up to the job. Certainly, we've had people say that keyring doesn't support their use case. But nor does pip - someone will need to write new code, and the idea of adding keyring support was to delegate handling this sort of use case to that project. Until we see a definite statement from them that they aren't interested in supporting the use cases being described here, there's not much pip should be doing (IMO). If keyring come back and say they don't want to support this use case, then pip needs to look at what to do - and in my view, I'd want to reconsider whether we should be looking for an alternative to keyring that does support our users - I still don't want pip to get into the business of credential management².

attempt to replace things that look like environment variables in configuration files

You can use environment variables in requirements files. Have you tried using that feature to see if it handles your use case?

¹ The approach of using keyring had the additional benefit of not requiring the pip developers to get into questions around what is a secure way of handling credentials - we could leave that to the experts maintaining the keyring project.
² Yes, environment variables seem like a simple enough solution, and safe enough. But I'm not an expert, so I'm not going to make that decision. That's basically the point here...

[lancetarn]

Thanks for your reply. I clearly only have a tiny fragment of the context here. I'll try to find an appropriate place to ask about keyring in CI. Also, I did not know that requirements files would expand environment variables. I had tried it within pip.conf, which does not appear to. I can probably get the job done through that mechanism!

I appreciate the perspective of leaving credential management to experts and your efforts to keep things going in the right direction.

[TimOrme]

@pfmoore chiming in here a bit and don't want to speak too much for others, but one of the cases mentioned elsewhere is that you end up in a bit of catch-22 situation that can't be resolved without some hacking, unless pip itself supports this.

If you're in an CI/CD environment where you only have access to a private, password protected PyPI repo, then you are in an unfortunate situation where you can't even install keyring to begin authenticating to that repo, even if it does support it.

There are perhaps other ways to get keyring installed, but they end up being a bit messy. Maybe an alternative is to ship keyring with pip or something along those lines, but I'm not sure of the feasibility or impact of that.

In short though, the concern is that if pip doesn't support that auth, and the only way to support auth is to install an external package, then we end up stuck in the cases where the external package requires auth.

[jasonstitt]

I think the "pushback" is because environment variables are a normal way of doing this and keyring is not. Several replies here have outlined the specific issues with adding this as a dependency. Given that, rather than requiring a "definite statement" from the keyring project (and who is going to obtain such a statement?), it would make more sense to explain how keyring is a good solution particularly for CI/CD, as it is basically an exception we would make from norm in order to use this tool. (i.e. this is the only similar tool that would use keyring...)

I still don't want pip to get into the business of credential management

Environment variables don't put you in the business of credential management. Something else is responsible for setting the environment variables; that's the point of environment variables.

Realistically the alternative here is not going to be keyring. An alternative is using PIP_INDEX_URL (an environment variable) with basic auth embedded in it. Which means you already take credentials in an environment variable, so the concern there is odd. And you already fixed the logging of the credentials in the URL several versions ago I think. The problem with this approach is simply that the entire URL now becomes a secret value, rather than just the credentials. I think this request is simply to split the URL from the credentials so that the URL could be hardcoded in a checkin without the credentials.

It sounds like environment expansion in requirements.txt is potentially superior.

[pfmoore]

@TimOrme

Maybe an alternative is to ship keyring with pip or something along those lines, but I'm not sure of the feasibility or impact of that.

You're right - bootstrapping keyring is an issue. But it was known (and acknowledged) when the feature was added, so all I can really say is that the original implementation saw that as an acceptable limitation. I don't personally have a good answer here.

Vendoring keyring is unfortunately not possible, because keyring depends on C extensions, and pip cannot vendor C extensions (because pip needs to be platform-neutral - there's a lot more background here, but that's the reality and it's not going to change, unfortunately).

@jasonstitt

and who is going to obtain such a statement?

Someone who needs this to work, surely? You seem to be assuming that it's up to the pip developers. Sorry, but it really isn't.

Which means you already take credentials in an environment variable, so the concern there is odd.

OK, I've no problem if you think my reluctance is odd. Feel free to take it as simply meaning that I won't do anything about this myself, if that helps.

It sounds like environment expansion in requirements.txt is potentially superior.

It does indeed sound like that is helpful for people in this situation. Which makes me wonder why no-one found that information. Is the section here in the documentation unclear? Is it hard to find? It may be that people have wasted time debating keyring, when if they'd found the existing feature they could have solved their problem much more easily - so if there's any improvement to the documentation that would have helped, it would be great if you could offer a suggestion (ideally as a PR, but even just an issue describing what you'd like to have seen would be good).

[lhupfeldt]

I guess that most people just put 'package>=version' (or == ...) in requirements.txt.
I definitely would never have had the idea to look at the requirements.txt specification in order to pass credentials.

Maybe some reference to requirements.txt from the existing documentation about how to authenticate, and an explanation of what to put in requirements.txt to make pip read credentials from there would help?

But I hope you are not referring to embedding credentials in index URLs or even adding index URLs in requirements.txt?
Credentials in URLs are generally considered insecure.

Adding URLs in requirements.txt for me would just make the file unreadable with even more substitutions to be made. We have production and not production pypi proxies, and I think other people will have the same.
It would also mean that every requirements.txt would have to add the credentials vs just having to add them on the CI server.

I understand that you are trying to limit the maintenance burden of pip, but we are talking 10 lines of code including logging (excluding the test) (and that would be maybe 7 if the check that both password and username is set was removed, as suggested) and 5 lines of documentation.

And this feature seems to be in popular demand.

[absassi]

@pfmoore you've said (emphasis mine):

Pip added support for keyring, in a good-faith attempt to handle the requests we were getting for a mechanism to store credentials outside of pip¹.

And I think this is the misunderstanding in this discussion. We aren't asking for a mechanism to store credentials outside pip. We already have that one (e.g. the credential store in our CI server), and our mechanism, whatever it is, provides the credentials in the form of environment variables (which is very common). However, this mechanism is not keyring. The problem we face is then: how do we pass these credentials, that are already in environment variables, to pip?

The option to make pip to use keyring directly is very nice and solves a valid, but different problem, which is how to take credentials from keyring and pass them to pip.

[imjohsep]

This probably isn't to everyone's standard, but I do this to store credentials as environment variables.