Pip: `pip install -r` does not expand environment variables in `requirements.txt`

[jorng]

• Pip version: 8.1.2
• Python version: 2.7.11
• Operating System: OS X 10.11.5
  

  Description:

When defining a requirements file using environment variables, installs fail with pip install -r. Issue does not occur when calling pip install directly

What I've run:

requirements.txt:

requests==$REQUESTS_VERSION

Output from pip install -r:

$ REQUESTS_VERSION=2.8
$ pip install -r requirements.txt 
Collecting requests==$REQUESTS_VERSION (from -r requirements.txt (line 1))
  Could not find a version that satisfies the requirement requests==$REQUESTS_VERSION (from -r requirements.txt (line 1)) (from versions: 0.2.0, 0.2.1, 0.2.2, 0.2.3, 0.2.4, 0.3.0, 0.3.1, 0.3.2, 0.3.3, 0.3.4, 0.4.0, 0.4.1, 0.5.0, 0.5.1, 0.6.0, 0.6.1, 0.6.2, 0.6.3, 0.6.4, 0.6.5, 0.6.6, 0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.7.4, 0.7.5, 0.7.6, 0.8.0, 0.8.1, 0.8.2, 0.8.3, 0.8.4, 0.8.5, 0.8.6, 0.8.7, 0.8.8, 0.8.9, 0.9.0, 0.9.1, 0.9.2, 0.9.3, 0.10.0, 0.10.1, 0.10.2, 0.10.3, 0.10.4, 0.10.6, 0.10.7, 0.10.8, 0.11.1, 0.11.2, 0.12.0, 0.12.1, 0.13.0, 0.13.1, 0.13.2, 0.13.3, 0.13.4, 0.13.5, 0.13.6, 0.13.7, 0.13.8, 0.13.9, 0.14.0, 0.14.1, 0.14.2, 1.0.0, 1.0.1, 1.0.2, 1.0.3, 1.0.4, 1.1.0, 1.2.0, 1.2.1, 1.2.2, 1.2.3, 2.0.0, 2.0.1, 2.1.0, 2.2.0, 2.2.1, 2.3.0, 2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.5.0, 2.5.1, 2.5.2, 2.5.3, 2.6.0, 2.6.1, 2.6.2, 2.7.0, 2.8.0, 2.8.1, 2.9.0, 2.9.1, 2.9.2, 2.10.0)
No matching distribution found for requests==$REQUESTS_VERSION (from -r requirements.txt (line 1))

Output when installing directly:

$ REQUESTS_VERSION=2.8
$ pip install requests==$REQUESTS_VERSION
Collecting requests==2.8
  Downloading requests-2.8.0-py2.py3-none-any.whl (476kB)
    100% |████████████████████████████████| 481kB 45.1MB/s 
Installing collected packages: requests
Successfully installed requests-2.8.0

[jorng]

Submitted fix in #3718

[sigmavirus24]

I don't think this is a bug. I doubt this functionality is documented anywhere, and while it may seem useful, this is absolutely outside the scope of a requirements file, in my opinion.

[jorng]

@sigmavirus24 A good reason this would be needed are for services like heroku, where it is recommended to store certain configuration in environment variables.

For example, it is not recommended to store a password in a git repo (in the requirements file), but if credentials can be read from an env variable file, it would allow for adding private repos in the requirements file.

[jamadden]

I tend to agree that this is out of scope. It seems to defeat
reproducibility, which is what a pinned version should be about, in my
opinion.

On May 24, 2016, at 19:56, Ian Cordasco [email protected] wrote:

I don't think this is a bug. I doubt this functionality is documented
anywhere, and while it may seem useful, this is absolutely outside the
scope of a requirements file, in my opinion.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
https://github.com/pypa/pip/issues/3717#issuecomment-221445208

[jayvdb]

When you invoke pip on the command line with requests==$REQUESTS_VERSION, it is your shell which is substituting in the value of the environment variable.

i.e. The program pip is invoked as pip install requests==2.8

Regarding private repos, you should be able to put the repository in your requirements, and store the authentication tokens in another file depending on the protocol (e.g. in ~/.ssh/ for ssh:// links), and that file should be stored in a secure manner. Storing cleartext passwords in env vars is begging for trouble.

[jorng]

@jamadden My original example may not be the best use for this. I actually think it is better suited for pulling credentials from environment variables, as mentioned above.

@jayvdb Since the requirements file syntax borrows from shell syntax (such as using # for comments), I am simply trying to continue that methodology.

I can agree that this may not be a bug, and instead be considered a new feature; but adding the feature would not hurt anything, and was quite simple to add (see #3718). Adding the feature would allow for better use of pip to resolve dependencies in python apps that are using the 12-factor app methodology.

As for the storing of credentials (including authentication tokens) in env vars, I defer to the 12-factor app methodology: http://12factor.net/config

Heroku has an example of using requirements.txt to download requirements from private repos, using:

git+https://user:[email protected]/nsa/secret.git

Ideally, though, one would not want to commit that to one's repository, especially if the repo is shared. A better way would be:

git+https://$GIT_USER:[email protected]/nsa/secret.git

Then any necessary credentials could be set as the env vars GIT_USER and GIT_PASSWORD

[RonnyPfannschmidt]

i would consider that kind of expansion the job of a actual template engine distinct from pip

my own opinion is , that adding this to pip now is mainly creating potential security issues for a convenience that should be in a different tool to begin with

[xavfernandez]

Cf #3514 for a similar PR/demand.

[jamadden]

but adding the feature would not hurt anything, and was quite simple to add (see #3718)

Similar to https://github.com/pypa/pip/issues/3717#issuecomment-221499106, I'd argue that this is not so simple and is best left externally. For example, the feature in #3718 is implemented with os.path.expandvars. However, that has differences depending on platform! On Windows, it will accept %var%, but not on posix systems, for example, thus making requirements.txt potentially non-portable.

[jorng]

@RonnyPfannschmidt I'm not sure what security issues you think it might cause, I actually think it would improve security for anyone using requirements files in this fashion.

@xavfernandez Good catch! I missed that PR. It appears to be implemented in a more cross-platform way, which is probably better, and addresses @jamadden concerns of non-portability.

I'll close this issue as duplicate and defer everyone to #3514