Conda-forge.github.io: Consider removing vc* "features"

[mbargull]

The imminent change to conda-build 3 could be good opportunity to remove

build:
  features:
    - vc9   # [win and py27]
    - vc10  # [win and py34]
    - vc14  # [win and py>=35]

from the docs and packages and only rely on vc version.* dependencies. Those vc dependencies should then be implicitly added through conda-build 3's combination of compiler jinja2 function plus run_exports from the new compiler packages. E.g., for some package that uses Cython with C++ extensions, replace

build:
  features:
    - vc9   # [win and py27]
    - vc10  # [win and py34]
    - vc14  # [win and py>=35]
requirements:
  build:
    - python
    - cython
    - vc 9   # [win and py27]
    - vc 10  # [win and py34]
    - vc 14  # [win and py>=35]
  run:
    - python
    - vc 9   # [win and py27]
    - vc 10  # [win and py34]
    - vc 14  # [win and py>=35]

by

requirements:
  build:
    - {{ compiler("cxx") }}
  host:
    - python
    - cython
  run:
    - python

[EDIT: Split build/host as per @jjhelmus's comment below.]
It does not really make sense to have those features -- i.e., expressions of strong preference, but not requirements -- when one already gives hard dependencies on a versioned vc package.

To me, it feels like "features" have been used to express hard dependencies though this is not really one of their capabilities. But I can only speculate on this as the introduction of "features" to conda-forge predates my involvement in the Conda ecosystem. Looking for references in this repository, I found:

• https://github.com/conda-forge/conda-forge.github.io/pull/302/files
• https://github.com/conda-forge/conda-forge.github.io/blob/master/minutes/2016-04-22.md
• https://github.com/conda/conda/wiki/VC-features/1e696b2606443eb882d8568e3b9c3bcaf46bba1c
• https://github.com/conda-forge/staged-recipes/wiki/VC-features/53b5448034d95b7007e0d5d8dbf04a2ac17bf197
• https://github.com/conda-forge/conda-forge.github.io/issues/64#issuecomment-204071904
• https://repo.continuum.io/pkgs/free/win-64/vc-14-0.tar.bz2 "Last modified: 2016-08-26 14:04:42"
• https://github.com/conda/conda-build/pull/582/files
• https://github.com/conda/conda-build/pull/600/files

Looking at the wiki pages' commit dates, it looks like "features" were introduced to conda-forge a couple of months before the vc packages have been created. That would explain why "features" were introduced in the first place, instead of simple/reliable dependencies on vc VERSION.*.

As is, a vcVERSION "feature" on a package that also depends on vc VERSION.* is redundant and additionally causes bad user experience due to differences/inconsistencies in conda's solver regarding the prioritization of channels/versions and "features" [EDIT] regarding explicitly and implicitly requested packages [/EDIT].

(cc @jakirkham, @ocefpaf, @msarahan, @mingwandroid, @mcg1969)

[mcg1969]

A thousand times yes! I fully support the removal of unnecessary features. Thanks for writing this up.

[ocefpaf]

Thanks @mbargull. This is already in the plans for the move to conda-build 3. Thanks to the awesome work that @msarahan did with conda-build 3 we will be cleaning up our recipes from a lot of clutter like that one.

Closing this just to reduce the noise here, but that is happening.

[mbargull]

Awesome, great to hear that! Thanks Mike and all!

[jjhelmus]

I'm in favor of this change and the plan to remove the vc features from conda-forge's recipes.
I just have two minor comments that are relevant to the discussion.

First it would be best to split the host and build sections in future recipes. So the pattern for Python packages with C++ Cython extensions would be:

requirements:
  build:
    - {{ compiler("cxx") }}
  host:
    - python
    - cython
  run:
    - python

Second, care needs to be taken so that the vc pinning is not to an exact version, vc ==14 but rather to any compatible versions vc 14.*. This is especially true for vc 14 as vc 14.1, the platform toolset version number for Visual Studio 2017, is compatible with Visual Studio 2015. Pinning to the exact version number unnecessarily prevents packages compiled with VS 2017 from being installed with packages compiled with VS 2015. The compiler activation packages in defaults, vs2015_win-64, etc, provide the compatible pinning as run_exports.

[msarahan]

Yes, as @jjhelmus mentions, recipes in the future just won't explicitly include vc at all. It's better to let the vs compiler activation package handle it with run_exports.

Jonathan's advice for not pinning vc exactly holds in the interim, until we can convert all recipes to depending on the run_exports behavior.

[ocefpaf]

Thanks. I'm starting a doc with the cb3 move info. Things are scattered in many issues and it is hard to keep track.

[dhirschfeld]

I'm starting a doc with the cb3 move info.

I'd be interested in reading that if there's a link?

[mingwandroid]

I have a few suggestions for your doc @ocefpaf, I've done some work on two common package build problems:

1. Overlinking (linking to libraries that are not actually needed), where -lsomelib is specified on the linker line but the library is not needed at all. For this, our compilers set -Wl,--as-needed, but you need a new libtool for this to work in any autotools based project (I will make a PR when you are ready).
2. Undeclared transitive dependencies. This is where a library in a newly built package has load commands / DT_NEEDED entries for libraries that are not direct run dependencies of this package. conda-build 3 has code now to either warn or error when this is discovered and I have a PR ready that implements a whitelisting capability (build/missing_dso_whitelist) for cases where missing libraries is expected (i.e. they are provided by the system).

Before any mass-rebuild commences it would be really good to make sure these things are in place.

[ocefpaf]

Thanks @mingwandroid!