conda 4.5.10 behavior change regarding solver selection of packages with features

[kalefranz]

In conda 4.5.9 via #7601 we added a feature flag to disable a solver rule that minimized the number of packages in an environment not having active features. That is, given that a feature is active in an environment, the rule sought to maximize the selection of packages that relied on that feature. For example, if debug was an active feature in an environment, the rule is designed to choose every debug package possible. This solver rule has been in place since conda 4.0.

Recently, the rule has led to issues with two isolated but widely impactful use cases.

Some six months ago, the Anaconda team patched defaults metadata to remove vc from the features key of packages, and instead use direct dependency relationships. This is the preferred pattern going forward. However, conda-forge (or any channel hosted on anaconda.org) does not have a workable method for patching package metadata short of removing and rebuilding the package. Therefore conda-forge is still using the old vc pattern whereby the vc version variant of a package is described in the features key. The result is that a change in channel priority, either moving from defaults to conda-forge or vice versa, will needlessly switch out the majority of the python packages in the environment.

While the vc problem affects only Windows users, users on all platforms are potentially affected by a similar problem regarding blas implementations. For example, see https://github.com/conda/conda/issues/7548.

The @conda/conda-core team has deliberated on this issue, and we are strongly considering releasing conda 4.5.10 with the feature flag featureless_minimization_disabled_feature_flag introduced in #7601 toggled from a default value of false to a default value of true. This will effectively change the default behavior of the solver regarding packages with features that has been present since conda 4.0. We do not make this decision lightly.

The alternative is to wait for the release of conda 4.6.0 to toggle the default value of this feature switch. In the last several years, the conda project has established a well-developed canary process in which major feature releases, as 4.6.0 will be, transition through beta and rc builds published to the conda-canary channel over a six-plus week period.

While the core team is still soliciting feedback, we feel that the accumulated urgency warrants swift action, justifying the behavior change in a rapid patch release.

To test the proposed conda 4.5.10 behavior for yourself, run conda info to ensure you're currently using conda 4.5.9, and execute

echo "featureless_minimization_disabled_feature_flag: true" >> ~/.condarc

The current plan is to publish conda 4.5.10 to the conda-canary channel today, followed by publication in defaults 48-72 hours later.

[dhirschfeld]

As one of the original reporters of the package shuffling problem on Windows I'm very much in favour of this change. I'll certainly give the new flag a go and see if I run into any problems...

[mcg1969]

I'm comfortable with this. It will break the "debug" use case, I'm afraid. But that's obviously a less important use case. We need to get features out of conda-forge packages in the long term, but for now this seems like the right solution. Of course I'd appreciate hearing from @dhirschfeld once he puts it through its paces.

[jjhelmus]

The setting of this configuration option does not seem to effect the replacements of the numpy package from conda-forge when installing a package (pandas) which depends on numpy.

Specifically after creating an environment with numpy from conda-forge, activating that environment, all attempts to install pandas results in the replacement of the conda-forge numpy package with a package from defaults

~$ conda info

     active environment : None
            shell level : 0
       user config file : /home/jhelmus/.condarc
 populated config files : /home/jhelmus/.condarc
          conda version : 4.5.9
    conda-build version : 3.10.9
         python version : 3.6.6.final.0
       base environment : /home/jhelmus/anaconda3  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/linux-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/pro/linux-64
                          https://repo.anaconda.com/pkgs/pro/noarch
          package cache : /home/jhelmus/anaconda3/pkgs
                          /home/jhelmus/.conda/pkgs
       envs directories : /home/jhelmus/anaconda3/envs
                          /home/jhelmus/.conda/envs
               platform : linux-64
             user-agent : conda/4.5.9 requests/2.18.4 CPython/3.6.6 Linux/4.13.0-45-generic ubuntu/16.04 glibc/2.23
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False
~$ conda create -n cf_numpy python=3.6 numpy -c conda-forge
Solving environment: done

## Package Plan ##

  environment location: /home/jhelmus/anaconda3/envs/cf_numpy

  added / updated specs: 
    - numpy
    - python=3.6


The following NEW packages will be INSTALLED:

    blas:            1.1-openblas                          conda-forge
    bzip2:           1.0.6-h470a237_2                      conda-forge
    ca-certificates: 2018.4.16-0                           conda-forge
    libffi:          3.2.1-3                               conda-forge
    libgcc-ng:       7.2.0-hdf63c60_3                      defaults   
    libgfortran:     3.0.0-1                               defaults   
    libstdcxx-ng:    7.2.0-hdf63c60_3                      defaults   
    ncurses:         6.1-hfc679d8_1                        conda-forge
    numpy:           1.15.0-py36_blas_openblashd3ea46f_200 conda-forge [blas_openblas]
    openblas:        0.2.20-8                              conda-forge
    openssl:         1.0.2o-h470a237_1                     conda-forge
    python:          3.6.6-h5001a0f_0                      conda-forge
    readline:        7.0-haf1bffa_1                        conda-forge
    sqlite:          3.24.0-h2f33b56_0                     conda-forge
    tk:              8.6.8-0                               conda-forge
    xz:              5.2.4-h470a237_1                      conda-forge
    zlib:            1.2.11-h470a237_3                     conda-forge

Proceed ([y]/n)? y

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use:
# > source activate cf_numpy
#
# To deactivate an active environment, use:
# > source deactivate
#
~$ source activate cf_numpy
~$ conda install pandas -c conda-forge
Solving environment: done

## Package Plan ##

  environment location: /home/jhelmus/anaconda3/envs/cf_numpy

  added / updated specs: 
    - pandas


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    six-1.11.0                 |           py36_1          21 KB  conda-forge
    pandas-0.23.4              |   py36hf8a1672_0        27.8 MB  conda-forge
    pytz-2018.5                |             py_0         193 KB  conda-forge
    mkl_random-1.0.1           |           py36_0         1.3 MB  conda-forge
    mkl_fft-1.0.5              |           py36_0         549 KB  conda-forge
    ------------------------------------------------------------
                                           Total:        29.8 MB

The following NEW packages will be INSTALLED:

    intel-openmp:    2018.0.3-0                            defaults   
    libgfortran-ng:  7.2.0-hdf63c60_3                      defaults   
    mkl:             2018.0.3-1                            defaults   
    mkl_fft:         1.0.5-py36_0                          conda-forge
    mkl_random:      1.0.1-py36_0                          conda-forge
    numpy-base:      1.15.0-py36h3dfced4_0                 defaults   
    pandas:          0.23.4-py36hf8a1672_0                 conda-forge
    python-dateutil: 2.7.3-py_0                            conda-forge
    pytz:            2018.5-py_0                           conda-forge
    six:             1.11.0-py36_1                         conda-forge

The following packages will be DOWNGRADED:

    blas:            1.1-openblas                          conda-forge --> 1.0-mkl               defaults
    numpy:           1.15.0-py36_blas_openblashd3ea46f_200 conda-forge [blas_openblas] --> 1.15.0-py36h1b885b7_0 defaults

[msarahan]

Collection of several related issues that we expect would be solved by this change:

• https://github.com/conda-forge/gdal-feedstock/issues/219
• Internal email thread titled "Reconsidering optimization condition for package count" starting on June 30
• Older libiconv package with feature being chosen over newer package without feature: https://github.com/conda-forge/freexl-feedstock/pull/9#issuecomment-401374853
• Phone discussions with Intel, where they see similar things as they try to remove their "intel" feature from packages
• https://github.com/conda-forge/fiona-feedstock/issues/91#issuecomment-411432355

Edit: unrelated to this exact issue, but other confusion with features:

• https://github.com/conda/conda/issues/7548
• Maybe, but might just be general feature confusion: https://github.com/conda/conda/issues/7454

CC @tomashek

[msarahan]

@jjhelmus I believe your use case is summarized superbly by @mbargull at https://github.com/conda/conda/issues/7454#issuecomment-400590772

[kalefranz]

• #6284
• https://github.com/conda/conda/issues/7299
• https://github.com/conda/conda/issues/6199

[jjhelmus]

From my limited testing setting featureless_minimization_disabled_feature_flag: true does not change the behavior of #7548 nor conda-forge/gdal-feedstock#219.

In both of these cases the solver minimizes the number of features in the environment by removing or excluding the conda-forg::numpy package. It is unclear to me if this is the expected behavior given the configuration.

[kalefranz]

I'm seeing the same things as @jjhelmus for #7548.

[dhirschfeld]

I have noticed an improvement in the package shuffling. It's hard to say exactly without testing over a longer period. I'm happy enough with it to leave it enabled in my .condarc

[dhirschfeld]

Regarding patching the metadata, would it not be possible to have a repodata.patch.json file which overwrites the data found in repodata.json and have conda always apply the patch file if present?

[msarahan]

@jjhelmus and I have discussed #7454 and #7548 separately. Jonathan tried to remove the code in the conda solver that minimizes the number of features. That then restores "normal" behavior. This is clearly not a workable solution, though, because we depend on that optimization to prioritize the MKL package such that it is installed by default whenever no specific blas metapackage is specified. Still, it seems highly unintuitive that this optimization should trump channel priority.

@mcg1969 can you please comment on thoughts behind this feature minimization taking precedence over channel priority? Is this intentional? Do we have use cases where we depend on this behavior?

[mcg1969]

Jonathan tried to remove the code in the conda solver that minimizes the number of features.

First of all, what _exactly_ did Jonathan do here? Did he remove the pass that minimizes the number of track_features present in the environment? I would _definitely_ veto that. That completely eliminates the ability to specify preferences between variants. That's a non-starter.

Or did he remove the lower-priority pass that minimizes the number of "feature mismatches"? This is the one I've been recommending disabling. This is the code that tries to make sure that if a track_feature is present, that for any packages with a variant that has the corresponding feature, it is included. That's the one I've been suggesting we experiment with bypassing, and it's the bypass that I believe has already been implemented.

can you please comment on thoughts behind this feature minimization taking precedence over channel priority? Is this intentional? Do we have use cases where we depend on this behavior?

The feature optimization passes occur _between_ requested package maximization and dependency maximization. That specific placement was indeed intentional. We iterated like crazy on that, unfortunately in response to considerable user pain.

I'm pretty sure we cannot move feature mismatch minimization below dependency channel priority maximization, because then it _also_ moves below dependency _change_ minimization. This would likely break the case where, say, someone installs nomkl after the fact.

_On the other hand_, Kale, @mbargull, and I have been discussing the notion of "strict" channel priority. In this mode, conda would not even _consider_ packages in a lower-priority channel if they exist in the higher-priority one. So for instance, for users that put conda-forge over defaults, conda would never reach into defaults for packages that live in both.

Kale's fear is that this will lead to more unsatisfiable solutions. I don't want to discount that fear completely but I am personally not concerned. I think we should implement it and try it out. My guess is that it solves this case and is far more benign than trying to rearrange features.

[mcg1969]

Kale showed me a proposed diff; assuming that's what you had in mind, it definitely _must not_ happen. For instance, if a user created an environment with defaults at the top channel priority, and then moved conda-forge to the top, conda would attempt to crossgrade every single package in then environment at its next opportunity. Can't do that, clearly.

[msarahan]

I am not at all proposing that we do that.

This is clearly not a workable solution, though, because we depend on that optimization to prioritize the MKL package such that it is installed by default whenever no specific blas metapackage is specified

"strict" channel priority sounds better, and I agree that we need to prioritize it highly. Users are suffering very badly from this, and we need a fix urgently.

[msarahan]

Would it be helpful for conda-forge to produce a numpy package that does not have features, so that perhaps it would be on equal footing solver-wise with our defaults numpy package?

[ocefpaf]

"strict" channel priority sounds better

IMO that is the only long term solution. Conda-forge can, and will, provide a no-features numpy but we are not given the freedom to do that whenever we can due to the brokenness of the state.

There are a number of other possible solutions, like reverting this behavior, then again it takes AnacondaInc freedom to roll this feature into conda.

From my point of view the only way for the two, or more, ecosystems to evolve together and benefit from each other would be the "strict" channel priority.

And just to be clear what I mean by strict channel priority means that conda will solve for the packages on the higher order channel only and use the lower order if, and only if, the packages requested are not available on the higher order channel.

I actually advocate for two more behaviors to be present on conda: the current one and a "package version priority" one. Of course all of those are not ensured to result in a stable environment but the user had the choice and will know how to debug their env under each different scenarios. (IMO only the strict one is easy to debug though.)

[jjhelmus]

I modified the solver to completely ignore features by disabling the track feature minimization step (via this commit). This was entirely an experiment to see how the solver would behave in this configuration, I am not proposing that this be considered further as it break all types of good behaviors.

On the other hand, Kale, @mbargull, and I have been discussing the notion of "strict" channel priority. In this mode, conda would not even consider packages in a lower-priority channel if they exist in the higher-priority one. So for instance, for users that put conda-forge over defaults, conda would never reach into defaults for packages that live in both.

I believe @ocefpaf has asked about something along these lines in the past. I can imaging a few cases where this would break but I suspect that such breakage might actually be desired.

For example, if the highest priority channel had a package spam which required eggs version 2.0 but only provided eggs versions 1.0, "strict" channel priority would result in an unsatisfiable error. This could be see as a good behavior as it alerts the channel maintainers about the lack of a eggs version 2.0 package. On the other hand if a lower priority channel had eggs 2.0 this "strict" channel priority is preventing this channel from fulfill the requirement.

Could the "strict" channel priority be expanded to exclude only packages with the same name AND versions in lower priority channels?

[mcg1969]

Yes, I can think of manufactured cases like that, but I would like to know how likely that will be _in real life_. The obvious case we have to consider is conda-forge vs. defaults. In that case, wouldn't conda-forge need to be responsible for ensuring that the version of eggs they host is compatible with the version of spam they host?

[mcg1969]

Hmm, on the other hand, maybe conda-forge is a good citizen and keeps spam and eggs up to date. But then someone puts defaults ahead in channel priority (why would anyone do that! :-)) and then defaults chooses to build eggs but not spam.

[jjhelmus]

How about the following for a more likely scenario:

A biochemistry research group published a domain specific package to their channel, fast_dna_analysis, it requires more common package not yet available in defaults or conda-forge, downloader. As time goes on downloader becomes quite popular and is added to defaults and conda-forge and gets lots of bug fixes. The research group never updates their version of downloaded as it is now part of defaults. With strict priority anyone who installs fast_dna_analysis with the research groups channel as the highest priority will get the older version of downloader rather than the newer version with bug fixes.

[ocefpaf]

I don't see the problem with that scenario @jjhelmus b/c people adding their channel to higher a order should know what they are doing! It is not default behavior and requires active action on the users side. (Also, I've been saying this since day 1 of conda, every time someone adds a 3rd party channel to their config they should get a warning about that they are doing and that "warrant is voided.")

Also, the other two behaviors I described would cover that: the status quo and the "higher version".

FYI my suggestion is a copy of my Linux repository policy, but they actually have a priorities levels that goes from 0-99! Where 0 is the package higher version only and 99 the strict repo priority. I don't think that conda needs that, only 0 (strict channel), 1 (current), 2 (higher package version) would suffice IMO.

[ocefpaf]

but I suspect that such breakage might actually be desired.

Yes! That would help people to debug what is wrong with their channel ecosystem much quicker.

But then someone puts defaults ahead in channel priority (why would anyone do that! :-)) and then defaults chooses to build eggs but not spam.

More people than you can imagine. At my day job I recommend that, actually a "poor's man" implementation of my suggestion where we use only defaults and the conda-forge::package syntax to emulate a strict priority and get what is not on defatuls from conda-forge. The reason is b/c, and I said that many times, conda-forge was always meant to be a place for latest and unstable/experimenta,l packages, while defaults should be our "stable" safe bet.

[ocefpaf]

Could the "strict" channel priority be expanded to exclude only packages with the same name AND versions in lower priority channels?

This would not help debug things and would help know if an ecosystem is self-consistent within itself. I rather not have that but instead the 3 option behavior (or at least 2, strict and current).

[kalefranz]

@ocefpaf Is "higher package version" for you equivalent to the current channel_priority: false?

[ocefpaf]

@ocefpaf Is "higher package version" for you equivalent to the current channel_priority: false?

I guess so. Never realized that :smile:

(In my defense I would never use it but I do know some people that would.)

[mbargull]

Sorry, I'm a little late to this "party". Only skimmed over the "strict" channel prio discussion so won't comment on this yet.
But for featureless_minimization_disabled_feature_flag you correctly concluded that this doesn't solve the conda-forge::numpy de-prioritization because of conda-forge::blas's track_features.
From https://github.com/conda/conda/issues/6284#issuecomment-408873953:

An experimental flag featureless_minimization_disabled_feature_flag has been added with which the features of packages can be completely ignored. This helps with issues like this one that happen on Windows due to packages from conda-forge that still have features: vc.
Note this does not change the behavior around track_features, i.e., issues around conda-forge::numpy and conda-forge::blas=*=openblas still remain until track_features is removed from conda-forge::blas=*=openblas.

[dhirschfeld]

Is there any way I can see channel pinnings? I think I've pinned numpy to defaults because I need the performance of mkl.

[kalefranz]

@dhirschfeld Use conda config --show-sources and also check for a file conda-meta/pinned in the prefix of your environment.

[kalefranz]

At this point, I think it's pretty clear that what we _thought_ was the solution to all of our problems, i.e. flipping the featureless_minimization_disabled_feature_flag flag, is in fact not the solution to all of our problems.

I therefore want to take a step back and make sure we actually understand the problem. I want to disentangle incorrect metadata from incorrect solver behavior, just so that we can confirm that what we believe to be _correct_ metadata is baseline okay.

I've created https://conda-static.anaconda.org/conda-forge, that has patched metadata as described by https://conda-static.anaconda.org/conda-forge/linux-64/patch_instructions.json. To test out all of the use cases and workflows we're referencing in this issue, use

conda config --set channel_alias https://conda-static.anaconda.org

and make sure it's just conda-forge and defaults in your channels list for now. Use any version of conda, and for conda 4.5.9 the featureless_minimization_disabled_feature_flag parameter can have the default value, which is false.

[mbargull]

Looks good so far for the track_features removal:

CONDA_CHANNEL_ALIAS=https://conda-static.anaconda.org CONDA_SUBDIR=win-64  conda search --info -c conda-forge \* --json | jq -r 'flatten|[.[]|select(has("track_features"))]|[.[]|[.track_features, .channel, .name, .version]|join(" ")]|sort|unique|.[]' | column -t 
anacondar     https://repo.anaconda.com/pkgs/r/win-64               _r-mutex                                              1.0.0         
blas_mkl      https://conda-static.anaconda.org/conda-forge/win-64  blas                                                  1.0           
blas_noblas   https://conda-static.anaconda.org/conda-forge/win-64  blas                                                  1.0           
clangcling    https://conda-static.anaconda.org/conda-forge/win-64  clangdev                                              3.9.1         
clangdefault  https://conda-static.anaconda.org/conda-forge/win-64  clangdev                                              3.9.1         
cling         https://conda-static.anaconda.org/conda-forge/win-64  cling-patches                                         1             
flang         https://conda-static.anaconda.org/conda-forge/noarch  flang-meta                                            0.0.1         
flang         https://conda-static.anaconda.org/conda-forge/win-64  flang                                                 5.0.0         
flang         https://conda-static.anaconda.org/conda-forge/win-64  flang_win-64                                          5.0.0         
flang         https://conda-static.anaconda.org/conda-forge/win-64  libflang                                              5.0.0         
mkl           https://repo.anaconda.com/pkgs/pro/win-64             accelerate                                            1.1.0         
mkl           https://repo.anaconda.com/pkgs/pro/win-64             accelerate                                            1.10.0        
mkl           https://repo.anaconda.com/pkgs/pro/win-64             accelerate                                            1.11.0        
mkl           https://repo.anaconda.com/pkgs/pro/win-64             accelerate                                            1.12.0        
mkl           https://repo.anaconda.com/pkgs/pro/win-64             accelerate                                            1.13.0        
mkl           https://repo.anaconda.com/pkgs/pro/win-64             accelerate                                            1.14.0        
mkl           https://repo.anaconda.com/pkgs/pro/win-64             accelerate                                            1.2.0         
mkl           https://repo.anaconda.com/pkgs/pro/win-64             accelerate                                            1.2.1         
mkl           https://repo.anaconda.com/pkgs/pro/win-64             accelerate                                            1.2.2         
mkl           https://repo.anaconda.com/pkgs/pro/win-64             accelerate                                            1.2.3         
mkl           https://repo.anaconda.com/pkgs/pro/win-64             accelerate                                            1.2.5         
mkl           https://repo.anaconda.com/pkgs/pro/win-64             accelerate                                            1.2.6         
mkl           https://repo.anaconda.com/pkgs/pro/win-64             accelerate                                            1.3.0         
mkl           https://repo.anaconda.com/pkgs/pro/win-64             accelerate                                            1.4.0         
mkl           https://repo.anaconda.com/pkgs/pro/win-64             accelerate                                            1.4.1         
mkl           https://repo.anaconda.com/pkgs/pro/win-64             accelerate                                            1.5.0         
mkl           https://repo.anaconda.com/pkgs/pro/win-64             accelerate                                            1.5.1         
mkl           https://repo.anaconda.com/pkgs/pro/win-64             accelerate                                            1.5.2         
mkl           https://repo.anaconda.com/pkgs/pro/win-64             accelerate                                            1.5.3         
mkl           https://repo.anaconda.com/pkgs/pro/win-64             accelerate                                            1.6.0         
mkl           https://repo.anaconda.com/pkgs/pro/win-64             accelerate                                            1.7.0         
mkl           https://repo.anaconda.com/pkgs/pro/win-64             accelerate                                            1.8.0         
mkl           https://repo.anaconda.com/pkgs/pro/win-64             accelerate                                            1.9.0         
mkl           https://repo.anaconda.com/pkgs/pro/win-64             accelerate                                            2.0           
mkl           https://repo.anaconda.com/pkgs/pro/win-64             accelerate                                            2.0.1         
mkl           https://repo.anaconda.com/pkgs/pro/win-64             mkl                                                   11.0          
mkl           https://repo.anaconda.com/pkgs/pro/win-64             mkl                                                   11.1          
nomkl         https://repo.anaconda.com/pkgs/main/win-64            nomkl                                                 2.0           
openjdk       https://conda-static.anaconda.org/conda-forge/win-64  openjdk                                               7.0.161       
openjdk       https://conda-static.anaconda.org/conda-forge/win-64  openjdk                                               8.0.112       
openjdk       https://conda-static.anaconda.org/conda-forge/win-64  openjdk                                               8.0.121       
openjdk       https://conda-static.anaconda.org/conda-forge/win-64  openjdk                                               8.0.144       
openjdk       https://repo.anaconda.com/pkgs/main/win-64            openjdk                                               8.0.152       
vc10          https://conda-static.anaconda.org/conda-forge/win-64  vc                                                    10            
vc10          https://repo.anaconda.com/pkgs/free/win-64            vc                                                    10            
vc14          https://conda-static.anaconda.org/conda-forge/win-64  vc                                                    14            
vc14          https://repo.anaconda.com/pkgs/free/win-64            vc                                                    14            
vc14          https://repo.anaconda.com/pkgs/main/win-64            vc                                                    14            
vc14          https://repo.anaconda.com/pkgs/main/win-64            vs2015_win-32                                         14.0.25123    
vc14          occt7.0.0                                             https://conda-static.anaconda.org/conda-forge/win-64  occt          7.0.0
vc14          occt7.1.0                                             https://conda-static.anaconda.org/conda-forge/win-64  occt          7.1.0
vc14          occt7.2.0                                             https://conda-static.anaconda.org/conda-forge/win-64  occt          7.2.0
vc9           https://conda-static.anaconda.org/conda-forge/win-64  vc                                                    9             
vc9           https://repo.anaconda.com/pkgs/free/win-64            vc                                                    9             
vc9           https://repo.anaconda.com/pkgs/main/win-64            vc                                                    9             
vc9           https://repo.anaconda.com/pkgs/main/win-64            vs2008_win-32                                         9.00.30729.1  
vc9           https://repo.anaconda.com/pkgs/main/win-64            vs2008_win-64                                         9.00.30729.1  

So that doesn't have blas[track_features=blas_openblas] and also, on Windows, python[track_features=vc*] anymore, which should cover the biggest chunk.
openjdk[track_features=openjdk] is another thing we have to get rid of -- not just on conda-forge, but also for defaults::openjdk=8.0.152.

I guess having track_features=vc* on the vc packages (apart from the newest/preferred one, i.e., vc=14.1) could make sense to prevent vc runtime switches for an environment. Though, I'm definitely not sold on this one as I'd rather avoid (track_)features stuff wherever possible. But I haven't given that special vc[track_features=vc*] case much thought yet..

[kalefranz]

After the patching at conda-static.anaconda.org/conda-forge, here are :

linux-64 packages with features

llvmcling:
  - clangdev-3.9.1-cling_1.tar.bz2
  - clangdev-3.9.1-cling_2.tar.bz2
cling:
  - clangdev-3.9.1-cling_3.tar.bz2
  - clangdev-3.9.1-cling_4.tar.bz2
  - cling-0.3.post-0.tar.bz2
  - cling-0.3.post-1.tar.bz2
  - cling-0.3.post-py27_cling_2.tar.bz2
  - cling-0.3.post-py35_cling_2.tar.bz2
  - cling-0.3.post-py36_cling_2.tar.bz2
  - llvmdev-3.9.1-cling_4.tar.bz2
  - llvmdev-4.0.0-cling_0.tar.bz2
llvmdefault:
  - clangdev-3.9.1-default_1.tar.bz2
  - clangdev-3.9.1-default_2.tar.bz2
flang:
  - clangdev-5.0.0-flang_0.tar.bz2
  - clangdev-5.0.0-flang_1.tar.bz2
  - clangdev-5.0.0-flang_2.tar.bz2
  - clangdev-5.0.0-flang_3.tar.bz2
occt7.0.0:
  - occt-7.0.0-occt7.0.0_0.tar.bz2
  - occt-7.0.0-occt7.0.0_1.tar.bz2
  - occt-7.0.0-occt7.0.0_2.tar.bz2
occt7.1.0:
  - occt-7.1.0-occt7.1.0_1.tar.bz2
occt7.2.0:
  - occt-7.2.0-occt7.2.0_0.tar.bz2
mesalib:
  - vtk-7.1.1-py27_mesalib_1.tar.bz2
  - vtk-7.1.1-py27_mesalib_2.tar.bz2
  - vtk-7.1.1-py35_mesalib_1.tar.bz2
  - vtk-7.1.1-py35_mesalib_2.tar.bz2
  - vtk-7.1.1-py36_mesalib_1.tar.bz2
  - vtk-7.1.1-py36_mesalib_2.tar.bz2
  - vtk-8.0.1-py27_mesalib_0.tar.bz2
  - vtk-8.0.1-py35_mesalib_0.tar.bz2
  - vtk-8.0.1-py36_mesalib_0.tar.bz2
  - vtk-8.1.0-py27_mesalib_0.tar.bz2
  - vtk-8.1.0-py27_mesalib_1.tar.bz2
  - vtk-8.1.0-py27_mesalib_2.tar.bz2
  - vtk-8.1.0-py27_mesalibh9c317cf_4.tar.bz2
  - vtk-8.1.0-py27_mesalibh9c317cf_5.tar.bz2
  - vtk-8.1.0-py27_mesalibhc26a1c2_3.tar.bz2
  - vtk-8.1.0-py27_mesalibhd4c0a90_2.tar.bz2
  - vtk-8.1.0-py35_mesalib_0.tar.bz2
  - vtk-8.1.0-py35_mesalib_1.tar.bz2
  - vtk-8.1.0-py35_mesalib_2.tar.bz2
  - vtk-8.1.0-py35_mesalibh9c317cf_4.tar.bz2
  - vtk-8.1.0-py35_mesalibh9c317cf_5.tar.bz2
  - vtk-8.1.0-py35_mesalibhc26a1c2_3.tar.bz2
  - vtk-8.1.0-py35_mesalibhd4c0a90_2.tar.bz2
  - vtk-8.1.0-py36_mesalib_0.tar.bz2
  - vtk-8.1.0-py36_mesalib_1.tar.bz2
  - vtk-8.1.0-py36_mesalib_2.tar.bz2
  - vtk-8.1.0-py36_mesalibh9c317cf_4.tar.bz2
  - vtk-8.1.0-py36_mesalibh9c317cf_5.tar.bz2
  - vtk-8.1.0-py36_mesalibhc26a1c2_3.tar.bz2
  - vtk-8.1.0-py36_mesalibhd4c0a90_2.tar.bz2
  - vtk-8.1.1-py27_mesalibh9c317cf_0.tar.bz2
  - vtk-8.1.1-py27_mesalibhf80e44b_1.tar.bz2
  - vtk-8.1.1-py27_mesalibhf884bf3_2.tar.bz2
  - vtk-8.1.1-py27_mesalibhf884bf3_3.tar.bz2
  - vtk-8.1.1-py35_mesalibh9c317cf_0.tar.bz2
  - vtk-8.1.1-py35_mesalibhf80e44b_1.tar.bz2
  - vtk-8.1.1-py35_mesalibhf884bf3_2.tar.bz2
  - vtk-8.1.1-py35_mesalibhf884bf3_3.tar.bz2
  - vtk-8.1.1-py36_mesalibh9c317cf_0.tar.bz2
  - vtk-8.1.1-py36_mesalibhf80e44b_1.tar.bz2
  - vtk-8.1.1-py36_mesalibhf884bf3_2.tar.bz2
  - vtk-8.1.1-py36_mesalibhf884bf3_3.tar.bz2

linux-64 packages with track_features

blas_mkl:
  - blas-1.0-mkl.tar.bz2
blas_noblas:
  - blas-1.0-noblas.tar.bz2
clangcling:
  - clangdev-3.9.1-cling_1.tar.bz2
  - clangdev-3.9.1-cling_2.tar.bz2
clangdefault:
  - clangdev-3.9.1-default_1.tar.bz2
  - clangdev-3.9.1-default_2.tar.bz2
cling:
  - cling-patches-1-0.tar.bz2
flang:
  - flang-5.0.0-1.tar.bz2
  - flang-5.0.0-20180101.tar.bz2
  - flang-5.0.0-20180207.tar.bz2
  - flang-5.0.0-20180208.tar.bz2
  - flang-5.0.0-hfc679d8_20180525.tar.bz2
  - flang_linux-64-5.0.0-20180525.tar.bz2
  - libflang-5.0.0-1.tar.bz2
  - libflang-5.0.0-20180101.tar.bz2
  - libflang-5.0.0-20180207.tar.bz2
  - libflang-5.0.0-20180208.tar.bz2
  - libflang-5.0.0-hfc679d8_20180525.tar.bz2
gf2x:
  - gf2x-1.1-1.tar.bz2
  - gf2x-1.2-1.tar.bz2
  - gf2x-1.2-hfc679d8_2.tar.bz2
mesalib:
  - mesalib-17.0.3-0.tar.bz2
  - mesalib-17.0.3-1.tar.bz2
  - mesalib-17.0.3-2.tar.bz2
  - mesalib-17.0.3-3.tar.bz2
  - mesalib-17.0.3-4.tar.bz2
  - mesalib-17.1.4-0.tar.bz2
  - mesalib-17.1.4-1.tar.bz2
  - mesalib-17.1.4-2.tar.bz2
  - mesalib-17.1.4-3.tar.bz2
  - mesalib-17.2.0-0.tar.bz2
  - mesalib-17.3.9-hdd5ec5b_0.tar.bz2
  - mesalib-7.6.1-0.tar.bz2
ntl:
  - ntl-10.3.0-2.tar.bz2
  - ntl-10.3.0-ha0399c4_3.tar.bz2
occt7.0.0:
  - occt-7.0.0-occt7.0.0_0.tar.bz2
  - occt-7.0.0-occt7.0.0_1.tar.bz2
  - occt-7.0.0-occt7.0.0_2.tar.bz2
occt7.1.0:
  - occt-7.1.0-occt7.1.0_1.tar.bz2
occt7.2.0:
  - occt-7.2.0-occt7.2.0_0.tar.bz2

[kalefranz]

And after patching, for conda-static.anaconda.org/main (equivalent to pkgs/main):

linux-64 packages with features

None!

linux-64 packages with track_features

nomkl:
  - blas-1.0-openblas.tar.bz2
  - nomkl-2.0-0.tar.bz2
  - nomkl-3.0-0.tar.bz2
openjdk:
  - openjdk-8.0.152-h46b5887_1.tar.bz2
  - openjdk-8.0.152-h6bedd7e_0.tar.bz2
rb241:
  - ruby-2.4.1-h8a22fef_0.tar.bz2
  - ruby-2.4.1-hd4a244d_0.tar.bz2
rb244:
  - ruby-2.4.4-hdc09616_0.tar.bz2
rb251:
  - ruby-2.5.1-h070849d_0.tar.bz2

[isuruf]

There are legitimate uses of features like flang feature where the clangdev package is pactched.

[kalefranz]

Right. And that's where disabling the solver rule, as originally proposed, will really start to mess things up I think.

[msarahan]

I'd clarify that there are legitimate uses for something like features, but features as a means of enforcing some ecosystem consistency are so deeply flawed that they should be eliminated. We currently depend on them for a sort of de-prioritization of packages, so that a given package collection (with the fewest features) is the default one that conda chooses.

There are better ways to do flang's stuff, and we'll try to help guide any transition. The blas and vc metapackages are the general scheme, but we could use some shortcuts to avoid things like "conda install blas=*=openblas numpy"

[kalefranz]

but we could use some shortcuts to avoid things like "conda install blas=*=openblas numpy"

Won't conda install openblas numpy work?

[msarahan]

Won't conda install openblas numpy work?

Sure, and that's roughly what we have in defaults with the nomkl package. Think of it as a metapackage that points to the specific "implementation mutex metapackage" - blas-*-openblas. Each thing that has multiple possible variants would need to create this scheme - implementation mutex packages, plus an optional metapackage to make it simpler to specify a given variant.

Openblas here isn't exactly the same idea, because it isn't a metapackage, but the net effect is the same.

There are legitimate uses of features like flang feature where the clangdev package is pactched.

For this use case, for example, there would need to be a clangdev implementation mutex package. Let's call it _clangdev_mutex. Let's have two versions of that metapackage:

• one for the default. Build string is "default"
• one for flang patched version (this one needs a track_feature, so that it is not the default). Build string is "flang"

clangdev needs to depend on one of these, depending on its implementation. People can now install the flang-enabled clangdev with an expression like:

conda install clangdev _clangdev_mutex=*=flang

but that's nasty. So we make a metapackage, say "flang", or even just the package "flang", and have it with a runtime dependency on _clangdev_mutex * flang

Now people can install the flang package either by:

conda install clangdev flang

or, if flang depends on clangdev itself, it could be just:

conda install flang

[isuruf]

What we currently have is that clangdev and _clangdev_mutex are merged into one package. So, we just need to change flang such that it doesn't use features and directly depends on the clangdev built with flang patches, right?

[msarahan]

The separate metapackage includes some critically important flexibility. If you don't have the metapackage, but you depend only on the dependency, then your pinning in flang on the clangdev package becomes unnecessarily tight, and you might also have to depend on crazy things like pinning to a specific build string with hash. The purpose of the metapackage is the mutex, but also to express the dependency portion of the hash in a more meaningful way.

[isuruf]

I revisited what I did and here's the summary.

• clangdev with flang patches has a features: flang which means it's not installed by default.
• flang package itself depends on clangdev and has track_features: flang

Is this correct behaviour?

[isuruf]

Hmm, I guess not if you consider that there can be 3 or more clangdev types.

[msarahan]

That's correct behavior for the use of features, but I'm discouraging use of features. What you could/should do instead is:

• clangdev package that has a dependency on a _clangdev_mutex-*-flang metapackage. May have a track_features entry to make it not the default when someone does conda install clangdev (without explicitly or implicitly specifying a _clangdev_mutex dependency).
• flang package depends on clangdev AND on _clangdev_mutex-*-flang. It does not have a track_features, because it has no competing installation candidates - there are no flang variants.

As long as every clangdev package has a _clangdev_mutex package dependency of some sort, this scheme will work. If any clangdev package is missing that dependency, conda is free to choose an incompatible older package that is less constrained. That's why hotfixing or replacing packages is so important here.

[isuruf]

Got it. Thanks.

[kalefranz]

Summarizing and underscoring the above discussion between @isuruf and @msarahan. In most all cases where features have been used historically, the ecosystem over the last year has been migrating to a different metadata pattern. Specifically, the features key in repodata should not be used at all. Rather than using the features key, we instead use a standard depends relationship on an appropriate base package. The track_features key is still used, but there is no longer any semantic meaning to "track features." The track_features key is now used to specify the "default" variant of a base package among a group of base packages. Specifically, what the channel chooses to be the "default" base package should not have a value set for the track_features key, and all of the non-default base packages should have a value set for the track_features key. The exact value used is irrelevant.

Let's look at the case of numpy, which has a dependency on a blas variant, for both defaults and conda-forge. For this use case, it's ok for multiple blas variants to be installed in the same environment; a different use case where only a single variant can be installed in an environment (like r-base / mro-base) has slightly different details.

In defaults, we want the default blas variant to be mkl. We also build numpy for openblas, but we don't want that to be the default. For historic reasons, our base variant packages are mkl and nomkl (the latter typically means openblas on linux, and on macOS will typically mean linking against the macOS accelerate framework in pkgs/free and openblas in pkgs/main). Every numpy package, which requires a blas implementation, should directly declare its blas dependency in the depends field. The mkl base package does NOT have a track_features field, since we want it to be the default variant for the channel. The nomkl base package does have a track_features field.

In conda-forge, we want the default blas variant to be openblas, but we also want the ability to build out packages for mkl, which gets pulled from defaults. There is a blas-1.0-openblas base package, which pulls in openblas as a dependency, being used as the variant base package. Every package that requires a blas implementation, and is built against openblas, should declare a dependency on blas=1.*=openblas. Every package built against mkl should just declare mkl as a dependency. The blas-1.0-openblas base package should not have a track_features key, since it's the default blas implementation for the channel.

I believe what I've described is correct, or very close. If there is anything inaccurate, let me know and I'll correct it.

[carlodri]

I might have just stumbled into an issue related to this: see https://github.com/conda-forge/pillow-feedstock/issues/45#issuecomment-412200802 and https://github.com/conda-forge/pillow-feedstock/issues/45#issuecomment-412273360

I'm not sure how to handle the problem...

and sorry, I couldn't follow all the details of this thread. If I did, I would probably not be asking...

[djhoese]

I'm not sure I'm following everything in this thread, but it sounds like anything that is currently not working between the defaults channel and conda-forge won't work until major changes happen in conda. Is that accurate?

My specific case is https://github.com/conda-forge/gdal-feedstock/issues/220 which has been linked previously and has gotten some attention from @ocefpaf and @msarahan. It is a simple case of wanting to use conda-forge to install my package's dependencies (gdal, numpy, etc) and run my unit tests. My package's releases have essentially stopped until I can get a working test environment on Travis/Appveyor or I guess I can merge fixes with failing tests and hope for the best. From what I understand the only fix for me as a regular conda user is to specify the exact version for every package and their dependencies. Is there anything else I can do so I can continue working?

Edit: Is there anything as a user and conda-forge package maintainer that I can do to help the progress of this?

[ocefpaf]

Edit: Is there anything as a user and conda-forge package maintainer that I can do to help the progress of this?

At the moment we have a poor's man strict channel, as a workaround until a better solution is in place. For that you can remove defaults (or conda-forge depending on you problem) and use only a single channel. On the conda-forge side we are temporarily vendorizing the defaults runtimes to allow for that solution to work.

(If you hit an unsolvable env with that solution it is better than a broken env b/c then we have something to work with.)

[djhoese]

@ocefpaf That seemed to work for me. For anyone else finding this and using astropy's ci-helpers, I did:

install:
    - git clone --depth 1 git://github.com/astropy/ci-helpers.git
    - source ci-helpers/travis/setup_conda.sh
    # See https://github.com/conda/conda/issues/7626#issuecomment-412922028
    - conda config --remove channels defaults
    - conda install -y $CONDA_DEPENDENCIES

And my tests pass.

[carlodri]

I am wondering if there is anything I can do to fix this PR: https://github.com/conda-forge/pycroscopy-feedstock/pull/4 which is failing (I think) due to the pillow package being wrongly pulled from defaults rather than conda-forge. I also tried to manually remove the defaults channel in AppVeyor but no luck, I get a lot of "package not found" errors.

I'm posting here since from this discussion https://github.com/conda-forge/pillow-feedstock/issues/45 (the issue I see in the CI) it appears that the problem could be related to this issue.

[msarahan]

closing this - strict channel priority is the right answer here, and is available in conda 4.6.

conda config --set channel_priority strict

@dzelge and potentially others, please be aware that your use of this flag may need to change.