Pip: refactor --no-install/--no-download/--download (and use mkdtemp build dirs)

[qwcode]

pip install --download -> pip download
pip install --no-install -> pip unpack
pip install --no-download -> get rid of it (see comments from @g2p below)

along with this, move to mkdtemp build dirs (instead of static ones) in virtualenvs and global installs.
I think the --no-install/--no-download "workflow" is what fueled the static build dirs pip has, which have caused so many problems.

[g2p]

pip install --no-download isn't very useful or predictable compared to either

pip install unpack-root/*/
pip install download-root/*.*

The flag requires a well-known build dir which doesn't seem terribly safe (symlink attacks?) and gives an opportunity for multiple processes by the same user to step on each other's toes. --no-download pkgname also still requires network access (pip will hit PyPI or similar), whereas pip install download-root/* at least addresses the offline use case.

IMHO it would be better to deprecate this flag, and not make any new references to it or to the current "stuff accumulates in a static build dir" behaviour. Separating the other two flags into their own commands would be pretty good, I approve (and pip unpack should take a directory argument like download does).

[qwcode]

@g2p, I just updated the description regarding dropping --no-download.
when a pull for this comes up for review, we could try to surface the original motivation for the --no-install/--no-download combo and make sure we're not losnig a valuable use case.

[dstufft]

What is the work flow that is supported by pip download, pip unpack?

[qwcode]

in the description I mentioned "workflow" in reference to the existing --no-install / --no-download.
I'm guessing they were for local edits to packages, and then installing from those local edits. not sure?

as for pip download it's just better conceptually than pip install --download.

and pip unpack is better conceptually than pip install --no-install. As for why pip should help people do this kind of unpacking, I'm not sure, except that pip always has.

[dstufft]

Well that's what I'm trying to explore, why should we have this feature. Primarily because I want to see if we should even have this, and if we should what the best form for this feature to exist as. As it is we're just kind of guessing and unless we identify an actual use case that we can decide on if we want to keep supporting that use case or not I think we should just remove it.

Perhaps we should ping the pip mailing list and distutils-sig and see if anyone speaks up in support of this workflow?

[qwcode]

after the refactor, we'll have the following:

• pip download: it helps fill up --find-link dirs for fast local installs.
• pip unpack: yes, this is debateable
• make sure pip install path/* works. valuable I think.

with these pieces, _IF_ anyone was using the old --no-install / --no-download "workflow", then they can do it with these pieces, without us having to make a big decision on whether that workflow has merit or not.

but yea, we can ask about the old workflow on the list, or also ask @ianb directly. it came from him I'm guessing.

[pfmoore]

For what it's worth, I use pip install --download (what will be pip download) a lot. But I think of it in terms of "get the sdist" - either to have a local copy, or to then unpack it to make some local modifications before building. Two consequences of this:

• I'd be surprised if pip download downloaded a wheel - I don't think that's possible, but I wanted to be explicit, I am always looking for an sdist when I use the command.
• pip download /path/to/local/dir doesn't work as I expect (it copies the unpacked local dir). I use this reasonably frequently, and it always confuses me.

I'd actually argue for a pip sdist command, working like pip wheel rather than the simple pip download. It seems to me that it would have more useful semantics.

I don't see any real need for pip unpack - I'd just use pip sdist and unpack it myself without thinking. Remembering that pip unpack existed would be more effort than just doing that :-)

[qwcode]

then unpack it to make some local modifications before building.

just curious. what modifications?
I'm guessing making modifications was the original motivation for pip install --no-install, which would theoretically be replaced by pip unpack. (and if not replaced, just removed)

I don't see any real need for pip unpack

_if_ there is a substantial reason for people to be making local edits prior to install, then pip unpack just makes it easier to do that across a number of requirements

[qwcode]

pip download /path/to/local/dir

this should just fail immediately IMO, with "you can't download a local directory"

[pfmoore]

just curious. what modifications?

Usually, it's been to set universal=True in setup.cfg to get universal wheels, things like that.

Other things have been fiddling with custom script generation to try to use entry points (and hence pip's script support) instead, that's more the sort of thing I'd normally use a local clone for, but in at least one case (docutils) they use Subversion and frankly, life's too short to bother with svn :-)

[quodlibetor]

FWIW I use pip install --no-install when I want to get all dependencies, recursively, for a package. Generally this is to make sure that all the dependencies exist within some local repository, and if they don't, then build them -- sometimes in an alternate format, like RPM.

[quodlibetor]

I could switch to unpacking the tar.gz/zip manually -- I obviously don't actually need the unpack functionality to be built in.

[ncoghlan]

Just a heads up that CPython currently relies on --no-install in the venv test suite: I use it to make sure that venv & ensurepip are ignoring the related pip configuration settings as expected. (That is, when ensurepip originally wasn't ignoring the global settings, it would fail to install pip and test_venv would fail, so the test case demonstrates that is no longer happening)

If --no-install goes away entirely, we'll need to redesign those tests to do something else that would break the environment if the settings were being processed.

[zroadhouse-rmn]

I use the "pip install --no-install --download" command at the tail end of my build process to help gather the sdists for all dependencies and include them in my deployment artifact. This enables me to ensure that the code I'm deploying is the same as the code I've built and tested.

download_dependencies ()
{
  echo "*** download_requirements ***"
  pip freeze | grep -vE "(wsm.|pygrametl|pyhive|pep8|pyflakes|pylint|nose|teamcity-|unittest2|coverage)" > build/dependencies.txt
  pip install -i $INDEX_URL --no-install --use-mirrors --ignore-installed --download=build/$DIST_NAME/packages -r build/dependencies.txt
  echo "*** download_requirements done. ***"
}

Then at deployment time, I use the following commands to extract my deployment artifact, create the virtualenv, and ensure that only the included packages are installed:

app_install_build()
{
    if [ ! -d "$APP_BUILD" ]
    then
        echo "Installing $APP_BUILD"

        # Un-pack the tgz
        tar zxf "$APP_BUILD.tgz"

        # Create the virtual environment
        virtualenv -p "$APP_PYTHON" "$APP_BUILD"
        source "$APP_BUILD/bin/activate"

        # Install the packages
        if [ -f "$APP_BUILD/requirements.txt" ]
        then
            # New style packaging using source distributions that is installable via pip
            pip install --find-links="file://$APP_HOME/$APP_BUILD/packages/" --no-index -r "$APP_BUILD/requirements.txt"
...

I'm looking at upgrading to a newer version of pip to pick up wheel support. How is this workflow handled by the new commands?

[Ivoz]

@zroadhouse-wsm Everything still works for pip 1.5, which allows you to build (with the wheel package) and install wheels. This refactoring will be for 1.6 or later, in the future.

[zroadhouse-rmn]

@Ivoz - thanks! Just want to make sure that the basic workflow of being able to download sdists via pip is preserved. We will be utilizing wheels as a mechanism to speed up deploys by caching the binary builds off packages on the target servers. I.e. our deployment artifacts produced by the build system are architecture neutral (sdists) and our deployments are fast (wheel).

[qwcode]

"pip install --no-install --download" command at the tail end of my build process
to help gather the sdists

btw, --no-install is not necessary here. --download does not install, and ignores what's already installed.

[Ivoz]

Btw, thinking out loud this could be a great avenue to do some refactoring? e.g, introduce a new codepath for pip download (and perhaps, pip unpack) rather than hooking into the old one, that is far easier to understand.

[qwcode]

to be clear, since the deprecations appeared for 1.5, I think 1.7 is the timeframe for starting on this work IMO, but yes, @Ivoz , refactoring could occur : )

[dstufft]

Well we should get pip download into 1.6 I think. So people have it there to be able to migrate to before the old way is gone.

[Ivoz]

@qwcode it was my impression that in 1.6 pip download could coexist with pip install --download? And you'd only remove the latter in a later minor.

[qwcode]

ok, sure, adding pip download, could happen now. I was thinking about the rest.

rather than hooking into the old one, that is far easier to understand.

just realize that in whatever refactor you might attempt,. both install and download do really share a lot of the same logic. both have to download archives, unpack, and walk dependencies.

[Ivoz]

Oh, definitely. So IMHO this would be a good waypoint into making progress on refactoring install as well.

[qwcode]

btw, if you haven't seen it already, see #1526

[auuron]

What should i use instead of pip install --build?

[qwcode]

@auuron although none of this has been implemented yet, the idea is that pip would start using random tmp dirs for building, i.e. no fixed location that a user could set with --build.

[auuron]

@qwcode good idea, because now i'm doing that by myself using --build...

So, i understand that --build will work until this will be implemented and i can keep using it? (its very important for me to keep every single build in a diffrend temporary directory)
Abother question is that i keep getting warnings about DEPRECATION, is it possible to turn it off? (or all warnings)

[qwcode]

@auuron yes, it will keep working. sorry, no good way atm to turn off deprecation warnings, other than using -qq, i.e. doubly quiet which will only log errors. http://www.pip-installer.org/en/latest/reference/pip.html#console-logging

[Nabellaleen]

I don't understand the --no-clean deprecation. It make me do a download + unpack instead of simply use install, because if a large package install fail (for example, a dependancy is missing and cannot be downloaded automatically, making me do a pip install dep1), I have to re-download it and re-unpack it.

[Ivoz]

@Nabellaleen you may wish to make use of --download-cache to cache installs.

[qwcode]

@Nabellaleen ditto on using --download-cache, but also see http://www.pip-installer.org/en/latest/user_guide.html#fast-local-installs

[clayg]

debugging a build failure on a new version of a dependency and the author of the libary needed the build logs - I very much needed --no-clean and it worked on pip 1.5.4 with a warning - please do not remove --no-clean (or document the alternative for this use case?)

Thanks!

[qwcode]

fair point @clayg , I marked it for deprecation along with the others because it was tied into the static build dir code, which will be going away, but I can see keeping it for the dynamic tmp build dirs. an alternative use case could be that one has to pip unpack the troublesome dependency and build/install it manually to do troubleshooting. I'll note in the description that the fate of --no-clean is TBD.

[dstufft]

I think keeping --no-clean makes sense, and maybe even --build.

[qwcode]

I'm a little concerned about pip leaving tmp dirs because someone is using --no-clean (separate from --build)

[dstufft]

I don't think there's anything to be concerned about, that's exactly what the end user is asking for by using that option.

[qwcode]

It feels odd to me for a --no-clean flag to leave _random_ tmp dirs. one thought is to have --build imply no cleaning, and just keep the --build flag for those wanting to troubleshoot or self-manage a build area.

[dstufft]

Eh, I don't see any reason to refuse to do --no-clean with a random tmp dir. It's not something I would probably do, but if someone wants to I don't see any reason. What's the downside?

[clayg]

well what is --build supposed to do? I ran install with in a virtualenv with --build ./build and there's nothing in ./build after install? --build ./build --no-clean seemed to get what I needed, but I'm still left needing --no-clean. pip 1.5.4 doesn't have an unpack command? But if there was a command that just downloads and inflates without actually attempting the install the way pip would have during install (and leaving artifacts left over for debugging) - it's probably not quite what this particular use case would be going after (does pip always just python setup.py install?).

I needed something like "hey package maintainer, on my funky distro no one uses anymore pip install xyz failed with this crazy gcc i don't even" 'ok, np, run pip install --magic-flag xyz and give me everything in ./magic-place-where-all-the-build-stuff-is'.

--no-clean seemed to fit the bill in this one case, pip even warned me when I tried to do the install again and it encountered a left over build directory!

I'll probably bow out at this point, as I don't think I can do much to help you guys figure out "the right thing" beyond advocating for the use case - which i'm guessing you grok pretty clearly. Good luck!

[CARocha]

Hi guys i try install GDAL but i need compiled the source i try this tuto http://linfiniti.com/2013/02/installing-python-gdal-into-a-python-virtualenv-in-osx/ but in my venv local with pip 1.5.5 not working anything this command not working pip install --no-download GDAL
this log
DEPRECATION: --no-install, --no-download, --build, and --no-clean are deprecated. See https://github.com/pypa/pip/issues/906.
Cleaning up...
Could not install requirement GDAL because source folder /Users/n/proyectos/django/personal/venvmapas/build/GDAL does not exist (perhaps --no-download was used without first running an equivalent install with --no-install?)
Storing debug log for failure in /Users/n/.pip/pip.log

i try with this
$pip --download gdal
Usage:
pip [options]
no such option: --download

nothing happen :( which is the best way for install GDAL in my virtualenv??
Cheers

[blueyed]

My use case for --build is that I want to prevent pip install to fail because of an existing build dir ("pip can't proceed with requirement 'foo' due to a pre-existing build directory. … Please delete it and try again.").

Therefore I set it to a specific directory and remove this in the deployment script.

Without --build I could remove the automatic build directory, assuming that there's still a root somewhere?!

Another use case for --build is using '/tmp' or some other location backed by tmpfs (RAM filesystem).

[andycasey]

My use case for --build is using /tmp because some old FORTRAN codes break down if the folder name is too long, which it is for the default site packages folder. Having 'random tmp directories' that are short (I think ~40 chars was the limit for the FORTRAN code in question) in length would resolve my situation.

[derekstavis]

If --build no longer works and tmpfs is not sufficiently large to build a package what should I do?

[g2p]

Set TMPDIR to a directory with free space.

[derekstavis]

@g2p Thanks! It would be nice to reference this in deprecation message :)

[qwcode]

@derekstavis --build still works. (#2060 raised some doubts, but it still works in 1.5.6 based on my testing)

you do need to use -no-clean if you want the build contents to remain , and not be cleaned every install.

[qwcode]

@derekstavis the only exception to it working that I know of is #804

[derekstavis]

@qwcode yeah, I can confirm the --build still works in pip 1.5.6

[derekstavis]

but when combined with --target gives a fatal error on already existing files in build dir that wasn't correctly cleaned.

[qwcode]

in hindsight, I think it was a mistake to deprecate --build and --no-clean. see #2062 about reversing this.

[qwcode]

@derekstavis there's some fixes to --target in develop that @pfmoore worked on. possibly fixed. not sure.

[derekstavis]

I have worked out --build and --no-clean, by setting TMPFS and PIP_DOWNLOAD_CACHE env variables, things that are less likely to change in the future. I don't think it's a mistake to deprecate as long as the alternatives are well documented.

[derekstavis]

@qwcode I will try to build develop branch and test my scripts again to see if my problems got fixed.

[qwcode]

fyi, --build and --no-clean are no longer deprecated (see PR #2064). The warnings will be updated in the next pip release. I've updated the description of this issue to exclude mention of these 2 options.

[msabramo]

What is the status of this? I came to this because of:

$ pip install --download=. --no-install foo
DEPRECATION: --no-install and --no-download are deprecated.
See https://github.com/pypa/pip/issues/906.

and now I'm just confused. :-(

$ pip download
ERROR: unknown command "download"

[qwcode]

they're marked with RemovedInPip7Warning, i.e. destined to be removed in pip v7
i.e. this can/should be worked on now...

[dstufft]

I'm going to go ahead and remove this now. I think the only open thing is moving pip install --download to pip download. I've captured this in #2643

[rbtcollins]

@msabramo pip install --download --no-install has been broken for ages - it was behaving identically to pip install --download. So you can just use pip install --download in all probability.

[asonnenschein]

This command in Python 3.4:

pip install --download psycopg2

Gives me this message instead of downloading psycopg2:

You must give at least one requirement to install (see "pip help install")

Any ideas?

[pfmoore]

--download takes a directory name, so that's what it's treating psycopg2 as. It's then reached the end of the arguments and hasn't seen any requirements to install. Hence the error.

[meejah]

FWIW, I was using --no-download after doing a peep install of dependencies to ensure that a final pip install --no-download -e . didn't go and download anything further (that won't get verified, of course, because no peep). Now I'm giving pip a bogus proxy instead ...

[ssbarnea]

Is there a way to emulate a --dry mode with current version of pip?