Pip: Determine what should be done about --(install|global)-option with Wheels

[dstufft]

Currently pip has --install-option and --global-option to make it possible to pass flags to the setup.py invocation. However these currently get ignored completely when installing from Wheel. This previously wasn't a big deal because commonly these options are only used for compiled packages and those often don't have Wheels on the platforms you might need these options on. However now in pip 7.0 via #2618 we now attempt to automatically build wheels for all projects. This means that, assuming we can build a wheel, these options are just completely ignored.

So what do we do with these options? Do we make their use imply --no-use-wheel? Do we attempt to convert them? Get rid of them?

[qwcode]

I had previously opened #1716 for this. I'll close that.

[qwcode]

the only extra tidbit from #1716 is just the awareness that install-options are recognized when placed in distutils config files.

[pfmoore]

The main use I've made of --install-option (actually --global-option) is the obscene hack:

pip wheel --global-option "--with-libyaml" --global-option "build_ext" --global-option "-DYAML_DECLARE_STATIC" --global-option "-LC:\Work\Scratch\pyyaml\yaml-0.1.5\win32\vs2008\Output\Release\lib" --global-option "-IC:\Work\Scratch\pyyaml\yaml-0.1.5\include"  .\PyYAML-3.10.tar.gz

This lets me add build options to the extension build (by injecting an explicit build_ext step into the setup command). The reason for doing this is to avoid having to download and edit the source distribution just to add some settings into setup.cfg.

Maybe having a means to add section.key=value items into the source archive's setup.cfg when doing the build would be a more general solution, that is explicitly a "build step" option.

Something like --build-setting [build_ext]define=YAML_DECLARE_STATIC, maybe? Then there's no implication that it has an effect when installing from wheel, it's _clearly_ a build-only option.

(Whether this would be viable depends on what uses other people have for --install-option and --global-option...)

[xavfernandez]

According to the recently merged https://github.com/pypa/pip/pull/2537 , people seem to have a need for these options.

An implied --no-use-wheel when --install-option or --global-option are used seems easy enough.

[rbtcollins]

So the problem (and its not unique to the wheel autobuilding) is scoping those options.

We can and should use the options from requirements file binding to pass through to the wheels. But the top level --install-option is only defined as applying to the thing being installed, isn't it? Or is it global?

Anyhow, if someone can define 'right' here, I'll happily write the code for it.

Remember too that folk can always use 'pip wheel' and then -f wheelhouse to get exactly what they want.

[dstufft]

Scoping is certainly a problem, and I've thought before that maybe we should remove those options and instead we should do something like --build-option:myproject="--bar" or something that handles the scoping.

There's also the question of what --install-option and --global-option even means in the future (tm) where we're trying to move to a situation where we _only_ ever install from wheels and the only thing we use a source distribution for is building a wheel. In that scenario there is nothing to pass a --install-option too (and ideally anything people are using it for we'd either explicitly say we don't support, or we'd get a better, generic option for) and --global-option doesn't really make much sense because if you're only invoking a source distribution to build a wheel there is no global, only build.

On the other hand, we might not want to define the API for --build-option util the metabuild system is done so we can implement it in terms of that (e.g. if the API for metabuild is CLI based, we'd want to pass --foo options, if it's Pythonic "call this function" based, we'd want some way to translate CLI opts into Python opts).

In the short term it might just make the most sense that using --install-option and/or --global-option disables the automatic wheel cache and then open up a new issue to deprecate and replace those with a --build-option.

[rbtcollins]

@dstufft and I discussed this on IRC, he says that the resolution is: "--install-option, --global-option, --build-option will all disable the consumption of wheels, and any autobuilding of wheels." And that follow on work can be done to assess what we really want/need longer term.

[qwcode]

the question of what --install-option and --global-option even means in the future (tm)
where we're trying to move to a situation where we only ever install from wheels

why doesn't --install-option make sense when installing from wheels?
regardless of the format, you might want to set different paths for the install.

feels like a loss for wheel installs to have less flexibility than sdists.
(see https://github.com/twang817/pip/commit/e3338f7b087a9aba7e9033aa9078971fc33fc547 as an example of a pip user hacking this themselves, and wanting us to add this support)

how would this get sorted out in the "longer term"?

[dstufft]

Because --install-option, --global-option, and --build-option is conceptually "pass this option to the setup.py execution", it's documented that way and I'm sure people are using it that way. When we're installing from a Wheel we have no setup.py to execute and thus nothing to pass options to. Further more as we move to the meta build system we (most likely) aren't even going to be guaranteed to have a setup.py that exists for even a source distribution.

Longer term I think that the way to solve this is something like that:

• Figure out what people are using --install-option for that makes sense at all for wheels (one example I've seen is to be able to direct scripts to a different location) and expose a top level pip flag that supports that same use case. When installing from a source distribution we'll pass those options to setup.py and when installing from a wheel we'll take them into consideration for computing paths.
• Figure out what people are using --global-option for, and if it still makes sense in the new world order. I don't know off the top of my head what people _are_ using this for, so I can't speak to what this would look like.
• Figure out the meta build system and what API is available for pip to call to pass options into the build system, once that is nailed down figure out a way that we can expose those build options to both pip wheel and pip install.

[dstufft]

In short, the basic problem with those options is that they are explicitly tied to executing setup.py and one of their use cases is for people being able to pass random arbitrary options to their setup.py invocations.

[qwcode]

these all seem like valid uses of install-options that could be applied to wheels.

 --home                               (Unix only) home directory to install
                                       under
  --install-base                       base installation directory (instead of
                                       --prefix or --home)
  --install-platbase                   base installation directory for
                                       platform-specific files (instead of --
                                       exec-prefix or --home)
  --root                               install everything relative to this
                                       alternate root directory
  --install-purelib                    installation directory for pure Python
                                       module distributions
  --install-platlib                    installation directory for non-pure
                                       module distributions
  --install-lib                        installation directory for all module
                                       distributions (overrides --install-
                                       purelib and --install-platlib)
  --install-headers                    installation directory for C/C++
                                       headers
  --install-scripts                    installation directory for Python
                                       scripts
  --install-data    

yes, these options are conceptually tied to setuptools, but don't have to be literally. we'd map them over to our distutils scheme utility.

for sure, it's weird to use setuptools options when there's no "setup.py" involved, but the reality is things are fractured right now, and maybe it's ok to break a conceptual line, to give people support for install flexibility in the mean time.

to be clear, I'm ok with the approach of ignoring them for wheels for now, but just making it clear, that we could do otherwise, if a lot of people wanted this.

[dstufft]

We already support --root, I'm not sure what --home does, and it seems logical to me to add the top level arguments there straight to pip install.

[qwcode]

yes, agreed, I'd like to see new top-level options that are setuptools-agnostic

[qwcode]

--home is for the "home scheme". it's similar to the user scheme, but free standing, and not on the python path by default

[dstufft]

Going to drop this off a release blocker because @rbtcollins fixed it so we don't autobuild wheels if these options are specified, so that removes the immediate need for this.

[asfaltboy]

I may be off here, but I think it make sense to also not build wheels if a setup.cfg includes any of the install-options mentioned above. I just had the unfortunate "switched to wheels stuff broke" scenario with this.

[BrenBarn]

Someone just asked a stackoverflow question about this: http://stackoverflow.com/questions/31979029/installing-a-whl-python-package-into-a-specific-directory-other-than-the-defaul

is the current state of affairs that it is not possible to install a wheel to a non-default location? That seems like a problem that should be remedied quickly.

[domenkozar]

I'd like to tell pip to install wheels at a certain prefix, which is currently not possible at all (there is --target but then there is no way to tell where to install scripts).

[dstufft]

--root?

Sent from my iPhone

On Nov 15, 2015, at 5:02 PM, Domen Kožar [email protected] wrote:

I'd like to tell pip to install wheels at a certain prefix, which is currently not possible at all (there is --target but then there is no way to tell where to install scripts).

—
Reply to this email directly or view it on GitHub.

[domenkozar]

That's not exactly working:

pip install *.whl --root=/nix/store/0nr8sy9bgngqi5ijk2izxxmlm4dmvx25-python2.7-pbr-1.8.1/
...
/nix/store/0nr8sy9bgngqi5ijk2izxxmlm4dmvx25-python2.7-pbr-1.8.1/nix/store/rfcsnszaw2jlkqdgg65h3phss8xa0mlp-python-2.7.10/bin/pbr
...

root and prefix are different, each meaning:

• prefix: what path goes before bin, lib, include, etc
• root: where to put all files including prefix (I think this option doesn't make much sense once you have prefix, but I guess it's around for legacy reasons)

[domenkozar]

Opened https://github.com/pypa/pip/pull/3252 to address --prefix

[matham]

I just ran into this. We're trying to use namespace packages as a way to separate our optional binary deps on Windows.

For each of the deps, the binary parts are installed to e.g. python/shared or python/include etc. Also each dep has a module that is installed to kivy/deps/module_name. Where kivy is the main project.

The problem is when the the root project, kivy, is not in site-packages but in an alternate location. I would need to tell wheel to install the namespace module to the alternate kivy place. These binary packages are wheels, so the pip stuff doesn't work.

Using --root doesn't work as it creates intermediate site-packages directories. Anyway, root installs both the python stuff and the bins to the new location, which is not what I wanted.

Although maybe this is an issue with namespace packages, and not wheel?

Maybe when building, wheel should have the option to specify which files are allowed to be located elsewhere and allow during the install to specify the location? This would get around the no existing setup.py problem as with this it may not be needed anymore for these kind of settings.

[LeoHuckvale]

+1 for "--install-scripts" with Wheels.

Sorry to add to the noise, but I thought I might share a use-case.

We have an application which we drive with a Python package (library and accompanying scripts). We want to ship the Python package (and updates) in a format which is clean (for distribution to customers), so a source distribution isn't an option. The library component installs fine into site-packages (which is what we want), but we need to install the scripts to a custom location, where our application looks for them.

[anntzer]

It would be nice to clarify the error message when the pip command invoked is actually pip wheel itself. Note that this use if explictly mentioned in the docs (even though discouraged) at https://pip.pypa.io/en/stable/reference/pip_wheel/#customising-the-build:

pip wheel --global-option bdist_ext --global-option -DFOO wheel

This currently outputs the nonsensical(?) warning

/usr/lib/python3.5/site-packages/pip/commands/wheel.py:126: UserWarning: Disabling all use of wheels due to the use of --build-options / --global-options / --install-options.

Obviously _this_ use of wheels is not (should not be?) disabled.

[pradyunsg]

@dstufft Has anything actionable come out of this discussion?

[pisymbol]

I too have now ran into this (or at least something similar).

I need 'wheel install-scripts' to be able to install to a custom path. Right now, it zonks the script wrapper always in '/usr/bin' without the ability to change the prefix path.

Is there really no way to have console scripts post wheel install to a custom path? I can certainly do this with eggs.

[JordonPhillips]

+1 to --install-scripts for wheels

[bsolomon1124]

The "Disabling all use of wheels due to the use of --build-options / --global-options / --install-options" when building a wheel is quite confusing. It probably should not exist at all when pip wheel is invoked.

[sbidoul]

The "Disabling all use of wheels due to the use of --build-options / --global-options / --install-options" when building a wheel is quite confusing. It probably should not exist at all when pip wheel is invoked.

I agree this is confusing. Actually --build-options / --global-options / --install-options implies --no-binary :all:. For pip wheel this means downloading and building everything from source. (and for pip install this additionally implies using the legacy setup.py install code path.)

[sbidoul]

I'm thinking about this in the context of #8102 (deprecating the setup.py install code path).

To progress with that deprecation, I think we need to decide between one of these two paths:

1. deprecating --install-option together with the setup.py install code path and asking users to switch to --build-option (and the setup.py bdist_wheel path).
2. deprecating all 3 --(global|install|bulid)-option, and moving foward with PEP517 config settings (#5771)

My current impression is that the road towards (2) is still quite long, and (1) could be a reasonable intermediary step with an alternative that we can support quite easily (i.e. replace --install-option with --build-option).

[pfmoore]

I agree. I think we should aim towards (2) - config settings are still relatively untried in practice, precisely because the toolchain doesn't support them yet - but I don't want improvements like #8102 to be blocked on it. So (1) as an immediate step sounds perfectly reasonable to me.

[davidtvs]

I think I just ran into this too. I'm trying to pass an argument to pip install that can be accessed in setup.py so that I can adjust the installation according to the parameter.
After googling a bit it seems like the way to do this is through --install-option or --global-option but now I see the warning UserWarning: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option. and the installation fails completely because it can no longer find some packages.

I would definitely not expect this behavior and would expect install options to be easily accessible in setup.py

[TylerGubala]

I ran into a similar issue where I want to pass a prebuilt directory to pip wheel to avoid building a Python extension that I have already built from sources on the machine somewhere.

What is the best way? I don't think this is working:

"${PYBIN}/pip" wheel --global-option="--bpy-prebuilt=/build/linux/bin" /blenderpy -v -w wheelhouse/