Pip: Need a way in requirements.txt to force installing a package

[vbabiy]

I would love a way to be able to say, //in a requirements file//, "install this package even if it's already installed somewhere." This can be forced on the command line with the --upgrade flag, but you can't do that in a req file.

The motivation here is that locally-installed packages in a virtualenv differ from globally-installed packages in at least one important respect: Executable scripts from global packages typically don't work as expected in a virtualenv, because the shebang line points to the wrong python interpreter, thus effectively bypassing the virtualenv.

So, if I provide a requirements file that includes a line like foo==1.2.3, my expectation is that users will be able to run scripts provided by foo-1.2.3. In reality, those scripts will work for those users who don't already have foo-1.2.3 installed, and likely NOT work for users who //do// already have it installed, and I'll get confusing bug reports because the user probably doesn't know that they already have it installed somewhere else, and neither do I.

Also, I can't reliably provide automation scripts which use such python scripts, even if I call my own python interpreter instead of relying on the shebang line, because I don't know if or where the script is installed, or even if there's multiple versions installed - in which case which foo isn't much help because I need the //right// one.

Right now the only workaround that I know of is to put "-e some-repo-url#egg=foo==1.2.3" in my requirements.txt. That forces installation. But this is kind of silly: I don't care about the source, and the motivation is not at all obvious from reading the requirements file, and it adds more dependencies on development headers if foo needs to build C extensions.

For a concrete example where this bit me, see
http://trac.pythonpaste.org/pythonpaste/ticket/458
.
It also just bit me today while trying to support a user who was failing to install a Django app:
http://groups.google.com/group/ebcode/tree/browse_frm/thread/a12098921d61cdc3/a020706cf3d1d162?rnum=11&_done=%2Fgroup%2Febcode%2Fbrowse_frm%2Fthread%2Fa12098921d61cdc3%3F#doc_2c9b22010a6e1457

---

• Bitbucket: https://bitbucket.org/ianb/pip/issue/218
• Originally Reported By:
• Originally Created At: 2011-02-18 21:59:33

[vbabiy]

Hmm. The other workaround, of course, is to just instruct people to use "pip
install -U -r requirements.txt". Obviously this doesn't allow you to decide
per-requirement.

---

Original Comment By: Carl Meyer

[vbabiy]

Right, and I'd really prefer it to be per-requirement, and bake it into the
req file rather than making the user remember a flag.

Another option would be for pip to get smart about scripts, and make local
copies with munged shebang lines, but that sounds horrible and insane if it's
even possible to find the scripts to copy.

---

Original Comment By: slinkp

[slinkp]

Also filed as a virtualenv bug, but it looks like there is nothing virtualenv can do about it:
https://github.com/pypa/virtualenv/issues/124

[slinkp]

Also noting that the right command-line workaround is probably -I, not -U.
I'd actually love to be able to put either of those in a requirements file, but -I is probably more important (if you just want "the latest" why are you putting that in a req file?)

[carljm]

Well, the "right" workaround for now is a little tricky. -I is ugly as a general recommendation because it really does pretend nothing else is installed, so it won't uninstall a previous version before installing a new one, which can lead to odd bugs if you are actually upgrading. If you know you've got a clean virtualenv and just want it to ignore global stuff, then -I is good. And -U also works fine, but only because it's buggy and reinstalls even if the same version is already installed :-) It still respects version-pinning, so there's no harm in using it with a fully version-pinned requirements file.

In any case, the _right_ fix here is a new option that we don't have yet: --ignore-global or some such spelling, which ignores anything that isn't in the site-packages you're about to install to, but still does proper uninstall-on-upgrade.

I'm not convinced this new argument needs to be usable per-line in a requirements file (that'll make it somewhat harder to implement, I think). I'm open to convincing use cases, but I think if we had this argument it could safely just be applied to the entire requirements file if you know that scripts might be involved - disk space is cheap :-) It should definitely be something you can put on its own line in a requirements file and have it apply to the whole file, like e.g. --index-url.

Pull requests welcome!

[slinkp]

Ah, thanks for the insight on -I.

-U is not buggy, that's just an undocumented feature :)

Disk space is cheap, but packages with C dependencies can be a big source of installation pain depending on how hard they are to build. In fact those are the only reason I don't normally tell people to use virtualenv --no-site-packages, and for the same reason I'd much rather have the per-line-in-requirements-file version of this proposed feature.

I like telling users to do apt-get install lxml python-gdal python-imaging numpy or their platform's equivalent. I don't want to go back to telling them to install gcc, libxslt, and a dozen other things that neither of us really cares about and whose names are sometimes hard to discover - and may even change between OS distribution releases (grrr thanks a lot for that, Ubuntu).

But then, if I ever needed a console script provided by a package with C extensions, I'd still be screwed :-

[carljm]

The -U bug definitely feels like a bug when you want to upgrade a big list of packages to latest versions, and most of them are already at the latest version, and some of them are big... of course, I almost never want to do that, but some people do. Or so I'm told.

So if you have people install big C stuff with OS packages, why do you need those things in the requirements file at all? I guess just for consistency, being sure it's there one way or another...

In any case, my hesitation about per-line is pretty much just implementation complexity. So if a pull request shows up on my doorstep with an implementation that lets it be set per-line, and isn't terribly ugly, I won't turn it down...

[kmike]

The information about package update strategy can be obviously baked into requirements file this way now:

# requirements.txt
-r base.txt
-r apps.txt

'base.txt' contain all those long-to-install packages that should usually be only installed once:

# base.txt
PIL
lxml

and the 'apps.txt' contains requirements that are updated more or less regular:

# apps.txt
some-small-package-1
some-small-package-2

And then: pip install -r base.txt (or pip install -r requirements.txt), pip install -U -r apps.txt

So is the original ticket about providing some syntax sugar for the example above?

[carljm]

So is the original ticket about providing some syntax sugar for the example above?

No, it's about providing a way to ensure that a package will actually be installed inside a virtualenv, even when the same package is already installed in global site-packages (and the virtualenv doesn't use --no-site-packages). Particularly for packages with scripts, where the globally-installed script won't run within the virtualenv.

[slinkp]

Hm, based on carljm's response, I think -U is _not_ the right feature to leverage for this: I don't really want to force download/install every time pip is run, I just care about forcing a local installation _once_ so that scripts work.

Maybe this would have to be a new command-line option too: --force-local/-F or something.
But that raises more questions about how it would be implemented and what it should do if you're not in a virtualenv (bail out with an error? do nothing?).

kmlke's suggestion might be an adequate workaround for me right now although it could get hairy if the order of installation matters (eg. what if a package in base.txt depends on a package we want to install with -U? For pip to get the right version installed, the latter needs to be installed during the same run, so we can't put it in apps.txt. Listing it in both files might work but, yuck.)

[carljm]

353 offers a reminder that the new flag here, --ignore-global or --force-local whatever we call it (it could even just be --local, since we already use that for freeze in a parallel way) needs to also respect --user and consider the user installation dir a distinct "local environment" much like a virtualenv.

[slinkp]

Also just noting, since I hadn't realized this, that "-I" applies to _everything_ -- including dependencies, not just the requested package(s). It really does mean pretend _nothing_ is installed.

[dzen]

As there is a way to specify an option for VCS, is should be possible to have

-I packagename>=package-version ?

[Garrett-R]

Any progress on this?

[dzen]

Take a look to contraints
https://pip.pypa.io/en/latest/user_guide.html#constraints-files

[mwarkentin]

+1 - we have a shared deployment library we use across ~14 applications. Each one has a deploy/ directory w/ a requirements.txt with a single line:

deploylib>=2.14.2,<3.0.0

We try to follow semver for this library, so we only need to update this file whenever there's a breaking change and we can handle that on a per-app basis.

In general though, we'd like our devs to use the latest 2.X version of the lib - however we need them to remember to do pip install -U -r requirements.txt for this to happen.

We could also add a wrapper script, or a Makefile, or similar, but there's no way of enforcing that they install packages that way instead of pip install -r requirements.txt which is pretty much muscle memory at this point.

Ideally we could just add --upgrade as the first line of the requirements file - initially thought we could do this when reading the docs, but noticed after trying that only specific pip options are supported within requirements.txt.

We may just remove the the requirements file altogether and provide a script / makefile to install via pip cli directly.

[dstufft]

Closing this, I believe that this feature is a bit of an antipattern.