Staged-recipes: Python install line standard

[msarahan]

Can we establish a standard for how we do the whole python setup.py dance?

Please vote by posting a comment with the corresponding number. Majority after 1 week wins?

1. python setup.py install --single-version-externally-managed --record=record.txt
2. python setup.py install --single-version-externally-managed --root=/
3. python setup.py install --old-and-unmanageable
4. python -m pip install --no-deps .
5. pip install --no-deps .

I give up on strikethrough. 4 is out, due to error (https://travis-ci.org/conda-forge/staged-recipes/jobs/127374916#L405)

[msarahan]

3 for me

[jakirkham]

I still can't find docs for 3 so am concerned that this vanishes at some point without warning. If somebody can show me in their docs, I might change my tune, but for now this is a no-go for me.

We have determined that 4 errors so also a no-go.

I'm not clear on whether 5 always plays nice with conda-build or not so am leery of this one.

1 or 2 are the recommended way of doing this (ref1 ref2). While verbose and annoying, it sadly seems to fit with the standard python packaging ecosystem. Maybe we can just accept this minor annoyance until the real annoyance (writing recipes for Python packages at all) is dealt with in an automated fashion ( https://github.com/conda-forge/conda-forge.github.io/issues/28 ) ( https://github.com/conda-forge/conda-forge.github.io/issues/51 ).

[ocefpaf]

I am with @jakirkham here. We have been using 1 for a while now. There are reason to use pip (see https://github.com/conda-forge/ipywidgets-feedstock/pull/3#discussion_r61851362), but it seems that it does not work all the time.

[frol]

I like the option 5 just because it is the shortest one and, in the end, we will get the same result as pip-users installing the package with regular pip install. However, I would vote for options 1 or 2 (I'm not sure what is the difference between them) because using another package manager (pip) while building conda packages does not sound like a great idea.

BTW, source code of setuptools asks to avoid --old-and-unmanageable option.

Also, --root options implies --single-version-externally-managed, so the option 2 can be less verbose, yet I would probably not skip --single-version-externally-managed anyway, just to be clear on what is going on.

I was trying to find a way to set those arguments from options 1 and 2 via environment variables, so Conda-Build can just set those by default, but I failed to find them. :(

[jakirkham]

BTW, source code of setuptools asks to avoid --old-and-unmanageable option.

Interesting note. Thanks for dredging that up, @frol. Not sure if that means it is deprecated, has bad behavior, or something else. FWIW, it survived the 21.0.0 release.

Also, --root options implies --single-version-externally-managed, so the option 2 can be less verbose, yet I would probably not skip --single-version-externally-managed anyway, just to be clear on what is going on.

That's nice. This is also backed up by their docs. Maybe --root=/ is an option? Thoughts on this, @msarahan?

I was trying to find a way to set those arguments from options 1 and 2 via environment variables, so Conda-Build can just set those by default, but I failed to find them. :(

That would really be ideal. Let's not give up hope.

[msarahan]

I would be very happy to have --root=/

I was also thinking about setting these automatically with jinja. I'm not sure how ready that is today, but some logic like:

if (script in build) and (setuptools in requirements["build"]):
(add the incantation)

[scopatz]

I was trying to find a way to set those arguments from options 1 and 2 via environment variables, so Conda-Build can just set those by default, but I failed to find them. :(

That would really be ideal. Let's not give up hope.

You can always have an environment variable that is PY_INSTALL_OPTS or something that then gets used on all setup.py lines or similar.

[jakirkham]

I would be very happy to have --root=/

Should we update the list?

[jakirkham]

I was also thinking about setting these automatically with jinja. I'm not sure how ready that is today, but some logic like:

if (script in build) and (setuptools in requirements["build"]):
(add the incantation)

Interesting. I have been wondering if we should just have some jinja module that we load up that has common magic. For instance, adding these arguments, setting standard compiler arguments, etc.

[jakirkham]

You can always have an environment variable that is PY_INSTALL_OPTS or something that then gets used on all setup.py lines or similar.

Good point. That would work.

[frol]

@scopatz Hmm, I cannot find anything but custom Makefiles inlining PY_INSTALL_OPTS themselves. Anyway, I have tried it myself and it didn't work for me at least with setuptools=20.7.0 available in anaconda repos. Can you provide a link to the source code or documentation on this environment variable?

[scopatz]

@frol there isn't any. I was just saying that conda-forge can make up and define an envvar that stores that standard install line (or part of it). This convention can then be used by all of the conda-forge recipes than can / do follow a standard install procedure. Obviously, this would need to be documented by conda-forge.

[scopatz]

That is, say conda-forge defined an envvar CF_PY_INSTALL_OPTS="--single-version-externally-managed --root=/", then the recipe could use this by simply having an install command:

python setup.py install ${CF_PY_INSTALL_OPTS}

or in the meta.yaml

build:
   cmd:
      - python setup.py install {{ CF_PY_INSTALL_OPTS }}

Or whatever it is. The recipe author does need to tie this in explicitly, but they already have to write the whole line explicitly now.

[msarahan]

yes, that's fine. It's only necessary when setuptools is used, though. It would be nice to have a catch-all, where people wouldn't need to learn when it is or is not necessary.

[msarahan]

Moreover - unfortunately, it breaks things when it is present, but not needed (no setuptools) =(

[frol]

@scopatz Oh, got it now. However, I would vote against this since this will require non-standard conda build command on localhost which is not immediately obvious. If anything like that is implemetned, it should be standard, so implemented in Conda-Build. Still, this workaround shouldn't be implemented since this is a hack, not a solution. A solution for Conda-Build might be an extra option in the build section, for example:

build:
  number:0
  python:
    setuptools: []

, where the [] value should be used for extra arguments to python setup.py install ....

Yet, I'm not sure whether this is a good idea to hard-code this one-liner, which is irrelevant for other languages/build systems.

[minrk]

I'd go for 5. 4 also works if --ignore-installed is added. I think 4 doesn't work as-is because python -m seems to add . to sys.path, whereas pip alone doesn't. That difference was awfully surprising.

The disadvantage of 1-3 is that they all assume setuptools is imported in setup.py, whereas pip should do the right thing in both cases (mainly by importing setuptools itself). If a package doesn't use setuptools, then you have to use setup.py install _without_ the extra args in the conda recipe, and then if it starts importing setuptools you have to make sure that the recipe changes the setup.py args at the same time. If you use pip, the command in the recipe wouldn't change.

[jakirkham]

I must say I do like the simplicity of 5 as long as we don't run into issues with conda-build and pip interacting funny.

[ocefpaf]

I am :+1: to option 5.

Another question is: should we add pip as a build requirement or is it always guaranteed to be there? I see that we are adding pip to be on the safe side, but I would love to reduce the boilerplate even more by removing it.

[minrk]

The likely sources of issues with pip that I can see are the package being found and installation skipped (this was the issue with 4, and avoided with --ignore-installed) or issues caused by the wheel build stage (avoided with --no-use-wheel).

So a more rigorous command might be:

pip install --no-use-wheel --ignore-installed --no-deps .

at which point pip is doing _very little_ beyond ensuring that setuptools is used with --single-version-externally-managed.

These extra arguments can be added on a case by case basis, if issues are encountered with particular packages, because they shouldn't generally make a difference.

You could, however, specify some or all of them in pip.conf during the build:

[install]
ignore-installed = true
no-dependencies = true
use-wheel = false

edit: fixed typos in config

I don't know if there's a great way to get pip.conf loaded during conda-build. Perhaps a conda package could do that.

[msarahan]

This seems core enough that we might want to build it right into conda/conda-build. It might make pip-installed packages work a little better with conda. Thanks for your insight, @minrk.

[jakirkham]

This is some great stuff @minrk. At this point, I feel like this discussion is turning into a subset of this issue ( https://github.com/conda-forge/staged-recipes/issues/535 ). Maybe we can discuss such a package over there (assuming we are liking this).

[minrk]

@ocefpaf as for whether pip needs to be explicitly included, I think it's up to the python package. I _believe_ the reason you get pip with python is that it is a dependency of the python package in the defaults channel. This has not always been the case. I don't think Python requiring pip is happening at the conda level, and it is provided by a separate conda package. That means that if you built a new Python recipe that didn't express a dependency on pip, you could have python without pip. At the same time, if you require that conda-forge recipes are built with the conda-forge python package (once it exists), you would be able to make assurances that python always comes with pip. I think an explicit dependency on pip in cases where it is used directly seems the simplest and safest choice, but it is quite likely to always remain redundant.

[jakirkham]

I _believe_ the reason you get pip with python is that it is a dependency of the python package in the defaults channel.

That is correct. It has caused some issues with the solver in the past, but yes you do get it automatically. Though I kind of prefer the "explicit is better than implicit" mantra personally.

[ocefpaf]

Thanks for the explanation @minrk. So TL;DR better safe than sorry. Let's keep it there.

[jakirkham]

Should add NumPy 1.11.0 recommends using pip install .. Of course one can tack on the needed arguments in our case. See the message it prints when installing with setup.py.

Running from numpy source directory.

Note: if you need reliable uninstall behavior, then install
with pip instead of using `setup.py install`:

  - `pip install .`       (from a git repo or downloaded source
                           release)
  - `pip install numpy`   (last Numpy release on PyPi)

[jakirkham]

I believe the reason you get pip with python is that it is a dependency of the python package in the defaults channel.

Also, as further FYI. This can be disabled by setting add_pip_as_python_dependency to false in the config file.

[jakirkham]

So, to try to simplify things for all 5 use cases listed. I have added a PR that tries to handle all this option setting behind the scenes. Combined with other proposed changes that will eliminate pip, setuptools, and wheel as python dependencies, this will hopefully make writing python recipes a bit simpler. Here is the PR ( https://github.com/conda-forge/staged-recipes/pull/642 ).

[pelson]

If there are no problems with 5 (like it meaning that pip also becomes a runtime dependency), I'd go for that over any of the others. If in doubt, I vote for 1 though.

[patricksnape]

I would also vote for 1 or 5.

[anguslees]

Fwiw, the reason I started looking at conda originally was to avoid various annoyances and silent failures I was experiencing with pip. I guess if we can keep it as a build-time dependency then that might limit the issues, but I feel a bit disappointed if conda ends up being basically a wrapper for pip (for python cases) :(

[patricksnape]

@anguslees Would you be able to expand a bit on these issues you were seeing? We are discussing this here because we want the most straightforward and solid way to handle pure Python packages - if you have evidence that pip might behave poorly then that may strike 5 as an option!

[anguslees]

@patricksnape: Nothing that I tracked down to something reproducible so I'm conscious that I'm spreading FUD here, which is not what I want to do :(

I found semi-frequent examples of dependencies that would fail to install (and exit non-zero, etc afaict), and yet pip would continue as if they had succeeded (believe the package was installed, and go on to exit 0).

There were other issues resolving differing versioned requirements on the same package from multiple dependencies, where pip would take the first(?) version restrictions and just ignore the others, or some similarly naive algorithm (I forget the details here too). I literally first heard about conda at the same python conference where a pip maintainer also described this dependency ordering issue in a talk (I had also experienced this first-hand), and concluded that slide with something like "doing it properly is algorithmicly expensive, so we might never fix this."

pip development has picked up substantially recently (or at least, the major version has incremented quickly), so it's possible these issues no longer exist.

[kalefranz]

I'm personally partial to 3, because that's the one I know will install files "flatly" and directly into site-packages. No janky .pth stuff to worry about, no putting python files in .egg directories, etc. etc.

WRT

BTW, source code of setuptools asks to avoid --old-and-unmanageable option.

The history shows that line hasn't been touched since 2005.
https://github.com/pypa/setuptools/blame/04d10ff025e1cbef7ec93a2008c930e856045c8a/setuptools/command/install.py#L19

I've also never found any legitimate justification for the "try not to use this" comment.

[minrk]

I'm personally partial to 3, because that's the one I know will install files "flatly" and directly into site-packages. No janky .pth stuff to worry about, no putting python files in .egg directories, etc. etc.

I believe all the proposals do flat egg and pth-free installations. Certainly 5 does.

@anguslees it sounds like the issues you encountered had to do with pip's dependency handling, which should not affect anything here, as pip is only used to install the recipe's own package, not pull in any dependencies.

[mjscosta]

Hi, just came across this ticket after trying to update a recipe, and performing a pull request.
The option 1. is not working, resulting in:

Package: nuitka-0.5.28.1-py27_0
source tree in: /feedstock_root/build_artefacts/nuitka_1510768150711/work/Nuitka-0.5.28.1
+ source /opt/conda/bin/activate /feedstock_root/build_artefacts/nuitka_1510768150711/_b_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place
+ python setup.py install --single-version-externally-managed --record record.txt
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

error: option --single-version-externally-managed not recognized

I'll update to option 5.

[jasongrout]

I'm also having the issue @mjscosta mentioned (error: option --single-version-externally-managed not recognized) at https://github.com/conda-forge/commonmark-feedstock/pull/6#issuecomment-354022181. I'll also update to option 5.

[isuruf]

@jasongrout, commonmark 0.5.4 uses distutils, so --single-version-externally-managed cannot be used.

[jasongrout]

@jasongrout, commonmark 0.5.4 uses distutils, so --single-version-externally-managed cannot be used.

Ah, thanks!

[nicoddemus]

Can this be closed then?

[jasongrout]

Can we make sure a recommended solution is added to the docs first, with whatever caveats someone should know?

[isuruf]

It's mentioned in the docs here, https://conda-forge.org/docs/meta.html#single-version-externally-managed, but that's not what the general consensus in this thread is.

[jasongrout]

@jasongrout, commonmark 0.5.4 uses distutils, so --single-version-externally-managed cannot be used.

@isuruf - I tried using script: "pip install -v --no-binary :all: --ignore-installed --no-deps ." in https://github.com/conda-forge/commonmark-feedstock/pull/6, but the result was a package with no python files.

Can you shed more light on your comment above about using distutils, and what I might be able to do? This python packaging situation has me really confused right now.

[ocefpaf]

@jasongrout I'm experimenting with python -m pip install --no-deps --ignore-installed . and I had success 99% of the time I used in in conda-forge. I wonder if we should add the --no-binary to that.

PS0: that does the :all: do?
PS1: for that command to work you need to specify pip in the dependencies. Unfortunately conda-forge drifted from python standards and does not bundle pip with python :unamused:

[jasongrout]

IIRC, --no-binary needs an argument: https://pip.pypa.io/en/stable/reference/pip_install/#cmdoption-no-binary

[jasongrout]

I tried using python -m pip ... that you had above, and see No module named pip. Did you have pip in your build requirements?

[ocefpaf]

Did you have pip in your build requirements?

Yep. See edit above.

[microe]

I am sure that my vote doesn't count for much and that this issue is very old, but I vote for 5, modified:
5a. pip install --no-deps --ignore-installed .

--ignore-installed helps improve the likelihood that conda-build will finish as it enforces reinstalling the module.

I also note that for pure python (noarch: python), that none of the setuptools options (1-3) will build a wheel and that wheels have technical advantages to eggs (like no egg-info entries in the package and true python universality).

[leycec]

Necro. :skull_and_crossbones:

This probably constitutes a separate issue entirely, but it might be helpful to note in official documentation that the conda skeleton pypi your_package_name command autopopulates the build/script key with 1. (setuptools), which now appears to be thankfully deprecated by community consensus. Since most of conda-forge's other documentation has been updated in favour of 5. (pip), it might be useful to add a brief note advising users running conda skeleton pypi to manually rewrite the generated meta.yml file to contain:

requirements:
  build:
    ...
    - pip

build:
  ...
  script: python -m pip install --no-deps --ignore-installed .

Setuptools is the Great Hell. pip is the Lesser Hell. pip wins.

Ideally, conda itself would be patched to autogenerate the above build configuration. To get that slow ball rolling, we might consider opening yet another conda-build issue. Given that there are 452 unresolved issues, however, I wouldn't hold my breath on a hasty resolution.

Until that far-fetched day, a note in official conda-forge documentation might suffice. Thanks, all!

[orodbhen]

This is almost certainly the wrong place to ask this, but as I can't seem to find documentation for it anywhere, I will anyways. So I apologize in advance.

At some point I started using --single-version-externally-managed --record record.txt, though I don't remember where I got that idea from. I can't find any mention of it in the conda-build documentation.

This worked for awhile, but has now stopped working. Instead of copying the python files into the temporary directory created by conda-build setuptools copies the files into the site-packages directory of my virtual environment. So it seems that it's ignoring the ${PREFIX} variable.

I wonder why none of this has been clearly documented yet in the conda docs. Or perhaps I'm looking int he wrong place. If I can figure this out, I would be happy to submit a MR to the docs myself. What is the currently recommended way to do this?

Update: Actually it looks like $PREFIX is getting changed somewhere, even though the log output says it's being set to the temp directory.

[nicoddemus]

It is documented in the conda-forge docs: https://conda-forge.org/docs/maintainer/adding_pkgs.html#use-pip 👍